CPYFRMIMPF - multiple files

[gwilburn]

I have quite a few processes where I retrieve a text file from an FTP site, use CPYFRMIMPF to get the data into a native database file, then run RPG pgms over it.
Until now, i have simply ensured that the file was a single file with the same filename.
This is all done through a scheduled CLP and FTP script
What i'm looking for is a way to do an MGET (get multiple files into a single folder on the IFS) and then copy what I get (one or more files) into my Physical file.
Any ideas?

[GLS400]

Re: CPYFRMIMPF - multiple files

Hi gwilburn:

After your mget......you can combine .txt and .csv files using a .bat

pseudo .bat code:

del mycombinedfile.txt
copy myfilea*.* + myfileb*.* + myfilec*.* mycombinedfile.txt
del myfilea*.*
del myfileb*.*
del myfilec*.*

alternative:

del mycombinedfile.txt
copy *.txt mycombinedfile.tmp
del *.txt
rnm mycombinedfile.tmp mycombinedfile.txt

CL

strpco
strpccmd pccmd('x:\path\mybat.bat') pause(*no)
cpyfrmimpf  frmstmf('/path/mycombinedfile.txt blaha blah blah

That assumes all the file you retrieve with the MGET are in the same format and start with the same prefix or end with the same file type.

Best of Luck
GLS

[K2r400]

Re: CPYFRMIMPF - multiple files

Before the CPYFRMIMPF you can merge all file in once with qshell (with the QSH clp command)

find '/home/Myfolder/' -name 'myfil*.csv'  
-exec sort -n {} >/home/Myfolder/myfileResult \;

In this example, all the file named MYFIL*.CSV in the directory are merge into a new file

[gwilburn]

Re: CPYFRMIMPF - multiple files

Thanks... great ideas. I'll give them a try. I was thinking along the lines of using CFGHTTPSCH to create a Source member containing a listing the folder contents. Then read the source member (RPGLE) and do a copy for each of the files in the list... never thought of using those approaches.

[gwilburn]

Re: CPYFRMIMPF - multiple files

Before the CPYFRMIMPF you can merge all file in once with qshell (with the QSH clp command)

find '/home/Myfolder/' -name 'myfil*.csv'  
-exec sort -n {} >/home/Myfolder/myfileResult \;

In this example, all the file named MYFIL*.CSV in the directory are merge into a new file

Ran this interactively with QSH command first... it created a file for me, but the file contained non-display characters when viewed with notepad.

What am I doing wrong?

[K2r400]

Re: CPYFRMIMPF - multiple files

No problem.
It's necessary to create the result file with the good CCSID (1252 for windows for example) before the operation (in the same statement with the touch operation) :

touch -C 1252 /home/Myfolder/myfileResult | find  
'/home/Myfolder/' -name 'myfil*.csv'  
-exec sort -n {} >/home/Myfolder/myfileResult \;

[K2r400]

Re: CPYFRMIMPF - multiple files

The same example from a command line or CLP :

QSH CMD('touch -C 1252 /home/Myfolder/myfileResult | find  
''/home/Myfolder/'' -name ''myfil*.csv''  
-exec sort -n {} >/home/Myfolder/myfileResult \;')

[gwilburn]

Re: CPYFRMIMPF - multiple files

The same example from a command line or CLP :

QSH CMD('touch -C 1252 /home/Myfolder/myfileResult | find  
''/home/Myfolder/'' -name ''myfil*.csv''  
-exec sort -n {} >/home/Myfolder/myfileResult \;')

Ok--- two months later, i got around to using this again. I have tried this commend interactively with 3 files in "MyFolder"... File1 has 74 rows, File2 has 1 row and File3 has 1 row. The command creates a 4th file containing data from the first file it found, but duplicates some of that data... it also includes data from the second file, but nothing from the third file. The total number of rows in the result file is 298.

what am i missing? here is the command i ran in qsh

touch -C 819 /nyr/orders.csv | find '/nyr/' -name 'us*.csv' -exec sort -n {} >/nyr/orders.csv \;

[DeadManWalks]

Re: CPYFRMIMPF - multiple files

PGM                                                                                         
                                                                                            
     Dcl     &LSCommand     *Char       (200    )                                           
     Dcl     &ThisLib       *Char       ( 10    )                                           
                                                                                            
                                                                                            
  /* ************************************************************************+              
      Delcare Directory List receiver file                                   +              
   * *************************************************************************/             
     Dclf    MYP01                                                                       
                                                                                            
                                                                                            
  /* ************************************************************************+              
      Look for any files on the MY folder.                              +              
   * *************************************************************************/             
     RtvObjD   MYP01  ObjType(*File) RtnLib(&ThisLib)                                   
     ClrPfm     MYP01                                                                    
                                                                                            
     CHGVAR     VAR(&LSCommand) VALUE('LS /My/ > +                                     
                                      /QSYS.LIB/' *TCAT &ThisLIb *TCAT +                    
                                      '.LIB/MYP01.FILE/MYP01.MBR')                    
     QSH CMD(&LSCOMMAND)                                                                    
                                                                                            
  /* ************************************************************************+                  
      Look through the Directory List and copy and process each file         +                  
   * *************************************************************************/                 
Loop:                                                                                           
     RcvF                                                                                       
       MonMsg   MsgId(CPF0864) Exec(Goto CmdLbl(Eoj))                                           
       CpyFrmStmf FromStmf('/My/' *Tcat &MYP01) +                                       
                  ToMbr('QSYS.LIB/' *TCAT &ThisLIb *TCAT '.LIB/MYP02.FILE/MYP02.MBR') +   
                  MbrOpt(*Replace)                                                              
                                                                                                
       Call     MYR01      /* loads into MY  files */                                 
                                                                                                
     GoTo       Loop                                                                            
                                                                                                
EOJ:                                                                                            
ENDPGM

[K2r400]

Re: CPYFRMIMPF - multiple files

touch -C 819 /nyr/orders.csv | find '/nyr/' -name 'us*.csv' -exec sort -n {} >/nyr/orders.csv \;

The find function search all us*.csv file in the /nyr/ folder and his sub-folder too.
Verify how many files are find under qsh with :

find '/nyr/' -name 'us*.csv'

[gwilburn]

Re: CPYFRMIMPF - multiple files

OK - after all this I decided to write a pgm that lists the contents of a folder and copies them one by one into a PF using CPYFRMIMPF, and then moves the files that were copied into a subfolder. I'm not completely finished, but it's basically working.

 *                                                                                                  
 *  Parameters:  Folder  :  '/abc/123'                                                              
 *               File    :  'OUTPUTPF'                                                              
 *               Delimter:  '|' or 'x05'                                                            
 *                                                                                                  
 ********************************************************************                               
 *                                                                                                  
Fifs_list  if   e             disk    rename(ifs_list:ifs_lstr) usropn                              
 *                                                                                                  
 * PARAMETER LIST                                                                                   
 *                                                                                                  
D EntryParms      pr                  extpgm('IFS001')                                              
D  pfldr                        80a   Const                                                         
D  pfile                        10a   Const                                                         
D  pdlm                         10a   Const                                                         
D EntryParms      pi                                                                                
D  pfldr                        80a   Const                                Folder                   
D  pfile                        10a   Const                                Physical File            
D  pdlm                         10a   Const                                Delimiter                
 *                                                                                                  
D doclist         c                   const('/qsys.lib/asthhobj.lib/ifs_+                           
D                                     list.file/ifs_list.mbr')                                      
 *                                                                                                  
D q               s              1a   inz('''')                                                     
 *                                                                                                  
D qcmdexc         pr                  extpgm('QCMDEXC')                                             
D  cmd                        3000a   const options(*varsize)                                       
D  cmdlen                       15p 5 const                                                         
 *                                                                                                  
D cmdstring       s            512a                                                                 
 *                                                                                                  
 *---------------------------------------------                                                     
 **                 M A I N                                                                         
 *---------------------------------------------                                                     
                                                                                                    
 /free                                                                                              
                                                                                                    
  cmdstring = 'cfghttpsch option(*crtdocl) doclist(' + q +                                          
               doclist + q +') ' + 'strdir(' + q +                                                  
               %trim(pfldr) + q + ') subtree(*none) pattern(''*'')';                                
                                                                                                    
  qcmdexc(cmdstring:%len(cmdstring));                                                               
                                                                                                    
  open ifs_list;                                                                                    
  read ifs_lstr;                                                                                    
  dow not %eof(ifs_list);                                                                           
    cmdstring = 'cpyfrmimpf fromstmf(' + q + %trim(srcdta) + q + ') +                               
                 tofile(' + %trim(pfile) + ') rcddlm(*crlf) +                                       
                 flddlm(' + q + %trim(pdlm) + q +') +                                               
                 datfmt(*usa) rplnullval(*flddft)';                                                 
    monitor;                                                                                        
     qcmdexc(cmdstring:%len(cmdstring));                                                            
     on-error;                                                                                      
     leave;                                                                                         
    endmon;                                                                                         
                                                                                                    
    cmdstring = 'mov obj(' + q + %trim(srcdta) + q + ') +                                           
                 todir(' + q + %trim(pfldr) + '/ordhist' + q +')';                                  
    monitor;                                                                                        
     qcmdexc(cmdstring:%len(cmdstring));                                                            
     on-error;                                                                                      
     leave;                                                                                         
    endmon;                                                                                         
                                                                                                    
    read ifs_lstr;                                                                                  
  enddo;                                                                                            
                                                                                                    
  close ifs_list;                                                                                   
  *inlr = *on;                                                                                      
                                                                                                    
 /end-free                                                                                          
****************** End of data *********************************************************************