ibmi-brunch-learn

Announcement

Collapse
No announcement yet.

Interactive SFTP to replace current FTP

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Interactive SFTP to replace current FTP

    Hi,

    I am working on an assignment to replace our current FTP connections with the new sFTP connections. A new user SFTPUSER is setup/configured on our 'IBM i' (V6R1) to access new sFTP site. I am able to login on green screen using SFTPUSER and get/put files from/to the sFTP site but when I am trying to do that using my profile DEVUSER, it says permission denied. (The length of teh actual user names are <= 8 characters)

    My admin suggested me to use the batch mode and submit my CL (with below commands) under SFTPUSER after logging in with profile DEVUSER. It worked! PS - These commands are based on Scott Klement's guide.

    DCL VAR(&USER) TYPE(*CHAR) LEN(10) +
    VALUE('SFTPUSER')

    DCL VAR(&HOST) TYPE(*CHAR) LEN(100) +
    VALUE('HOST')

    DCL VAR(&CMD) TYPE(*CHAR) LEN(500)

    ADDENVVAR ENVVAR(SFTP_USER) VALUE(&USER) REPLACE(*YES)

    ADDENVVAR ENVVAR(SFTP_HOST) VALUE(&HOST) REPLACE(*YES)

    CHGVAR VAR(&CMD) VALUE('PATH=$PATH:/QOpenSys/usr/bin && +
    sftp -b /home/sftpuser/sftpuser.sftp $SFTP_USER@$SFTP_HOST')

    ADDENVVAR ENVVAR(QIBM_QSH_CMD_OUTPUT) +
    VALUE('FILE=/tmp/sftplog.txt') +

    QSH CMD(&CMD)
    MONMSG MSGID(QSH0000) EXEC(DO)
    SNDMSG MSG('File transfer failed! See /tmp/sftplog.txt') +
    TOUSR(DEVUSER)

    sftpuser.sftp scripts
    ************Beginning of data**************
    cd /inbox
    ls *.csv
    ************End of Data********************

    The above method is run in batch to execute the scripts. I want to know a way to run these scripts (and some more of them) one by one Interactively under any user including SFTPUSER.

    Is it doable? I tried many things but not able to come right with it. It's really urgent. Please help.

    Thanks

  • #2
    As far as I'm aware, the only way to do this would be to give all users access to the sftp profiles keys. I think that as you're doing the sftp, that would mean the IBM i is using the private keys so you would need to give public access to them. That is not something I'd EVER recommend! They're private for a reason.

    A better alternative would be to create a process whereby the person submits a batch job under the sftp profile, through adopted authority (I think that will work) or the swap user APIs.

    Comment


    • #3
      Thanks John.

      1) For running it in batch with adopted authority like I am currently doing, how can I make my script batch file sftpyser.sftp parameterised? For example, for one stream I may need to list all .csv files while for other I may do it for .txt files using the same script file. Or it's always one batch file per process?

      2) I shared the details of SWAP user APIs with my Admin and he wasn't sounding very convinced with the idea which is making me think that it would be a no no


      Comment


      • #4
        When I mentioned adopted authority, I was meaning using adopted authority to submit a job under the sftp user ID. Adopted authority is not honoured in the /Root filesystem so you can't use that to give a job access to the keys, your job needs to be running as the sftp user ID. The swap APIs were another alternative that would cause the job to run as the swapped-to user ID. The API however does need a bit of thought into controlling its use to prevent users deliberately or inadvertently swapping to the profile which may be why the admin wasn't keen on the idea...

        For the sftp scripts, it's hard to say without knowing what you are trying to achieve. If your requirements are either all csv files or all text files, you could have 2 scripts and have your program select the appropriate one. If that's not the case or you want better control, you would need to write a program to build the sftp script on the fly based on your input parameters.

        Comment


        • #5
          You say you are getting "permission denied". Can you give us a bit more information? Permission is denied to WHAT?

          Please keep in mind that there are TWO userids involved... the one on your IBM i server where your script runs, and the one on the SFTP server you're connecting to. The &USER variable in your CL program is for the SFTP server, NOT for the IBM i system you're running the program from. I want to make sure that's clear, because you say you're submitting the job under the same userid (SFTPUSER) that you specified in the CL program, and there's absolutely no requirement that these names match since they pertain to different systems. What is important is that the IBM i user running the script has adequate authority to do the things required (read the script file, save the downloaded files, etc) and has their ssh keys set up properly. And on the sftp server side, the userid matches what you've given for &USER and has the public key installed. (There can be many public keys if multiple remote users are logging on to the server with a single profile.)

          With regards to keys, it's my opinion that each user who runs the CL program on the IBM i should have their own keys generated via the OpenSSH tools, and all of their public keys should be installed on the appropriate userid on the SFTP server. This way, there's no security problem that goes with sharing keys because each user has their own. (Of course, whomever is running the SFTP server must agree with that approach.)

          If you do want to share a single key for all of your IBM i CL program users, you can do so by specifying it on the "sftp" command. Right now you have something like "sftp -b /path/to/script user@host", but you could change it to "sftp -oIdentityFile=/path/to/id_rsa_file -b /path/to/script user@host". This would allow you to generate the ssh key and put it in a public place where it is shared with all users without it being installed in a particular user's home directory

          Comment


          • #6
            Thanks John & Scott.

            I am getting below error when I try sFTP Interactively or in batch using my user id (which is different from the SFTPUSER id). For your reference, only one user (SFTPUSER) is created on the sFTP server.

            Permission denied (publickey,password,keyboard-interactive).
            Connection closed
            $

            I think it's related to authority and the key setup for my profile as you explained above.

            Many thanks for your help with this. Hopefully I get this right this time. If not, I will come back here. Thanks again.

            Comment


            • #7
              All the error means is that none of the authentication methods worked. (publickey is the one you're trying, but it also tried password and keyboard interactive, which won't work because you're not using a real Unix terminal here, so it thinks you're "in batch" rather than interactive.)

              It could be the authority problem, or could be that the key isn't installed/recnogized properly on the server.

              Comment


              • #8
                You are right Scott. After your previous post, I checked with my admin. The key was not installed on the server. Admins are still analysing if it's the best option to install different users keys on the sFTP server. I will keep you posted with further progress on that. While I am waiting for them to come back, I am going to program it for batch and will use John's idea to write new scripts in the script file run time to build and run the scripts dynamically under the SFTPUSER as suggested by you. Many thanks for your help.

                Comment


                • dhirajt30@gmail
                  dhirajt30@gmail commented
                  Editing a comment
                  Hi Scott...hope you are good.

                  Sorry I am back again.

                  As I said earlier, I am retrieving the sFTP .csv files (Pipe delimited) using below scripts (Batch mode):

                  CHGVAR VAR(&CMD) VALUE('PATH=$PATH:/QOpenSys/usr/bin && +
                  sftp -b /home/sftpuser/sftpuser.sftp $SFTP_USER@$SFTP_HOST')

                  The files in IFS are copied with EoL *LF while when I am fetching the same file using FTP (Binary, namefmt 1), they have *CRLF EoL.

                  I understand that it may be something to do with Binary or ASCII but not sure how can I retrieve my sFTP files with *CRLF EoL.

                  Can you perhaps pls help.

                  Thanks.

              • #9
                Hi, I really need help with this. Any qshell script to convert LF to CRLF (IFS Csv file) would also do. Thanks.

                Comment


                • #10
                  I'm not aware of any way to tell SFTP to convert from LF to CRLF. SFTP simply copies the data, it does not get into the business of transforming it to other formats (as the old FTP did, to some extent.)

                  If you are reading the file with CPYFRMIMPF or CPYFRMSTMF after the transfer, just change your command to use *LF instead of *CRLF and you're all set.

                  If you're reading the file with the fopen() and fgets() API, then add the CRLN option to the fopen() so that it does not expect a CR

                  If you have no control over the application that reads the file, then just convert it. One easy way is to use "sed" in a Qshell script like this:
                  Code:
                  #!/usr/bin/qsh
                  eval CR=`printf \r`
                  sed "s/$/&$CR/g"
                  You can call that script as follows
                  [code]
                  /path/to/THESCRIPTNAME < inputfilename > outputfilename
                  [/code

                  Comment


                  • #11
                    Thanks Scott.

                    Comment

                    Working...
                    X