sFTP and continuation character

[RobertC]

I found a PowerPoint presentation you did titled ''Scripting the OpenSSH, SFTP, and SCP Utilities on i''. As my guide I used the sample CL program from the ''Running Expect from a CL Program'' section. Here is my program with the user name, host name and password x'ed out. I have included the corresponding script and the expect log as well.

PGM
DCL VAR(&USER) TYPE(*CHAR) LEN(17) +
VALUE('accountxxxxxxxxxx')
DCL VAR(&PASS) TYPE(*CHAR) LEN(5) VALUE('xxxxx')
DCL VAR(&HOST) TYPE(*CHAR) LEN(12) +
VALUE('xxxx.xxxx.xx')
DCL VAR(&CMD) TYPE(*CHAR) LEN(500)
CHGVAR VAR(&CMD) +
VALUE('PATH=$PATH:/Qopensys/usr/bin:/usr/lo+
cal/bin && expect -f myscript.exp')

ADDENVVAR ENVVAR(SSH_USER) VALUE(&USER) REPLACE(*YES)
ADDENVVAR ENVVAR(SSH_HOST) VALUE(&HOST) REPLACE(*YES)
ADDENVVAR ENVVAR(SSH_PASS) VALUE(&PASS) REPLACE(*YES)

ADDENVVAR ENVVAR(QIBM_QSH_CMD_OUTPUT) +
VALUE('FILE=/tmp/expect.log') REPLACE(*YES)

ADDENVVAR ENVVAR(QIBM_QSH_CMD_ESCAPE) +
VALUE('Y') REPLACE(*YES)
QSH CMD(&CMD)
MONMSG MSGID(QSH0000) EXEC(DO)
SNDMSG MSG('File Transfer failed! See +
/tmp.expect.log') TOUSR(ROBERT)
ENDDO
ENDPGM


Here is the corresponding script. I hardcoded the user and host but plan to use the environment variables once I get the file transfer to work.

#!/usr/local/bin/expect -f
set timeout 20
spawn sftp accountxxxxxxxxxxx@xxxx.xxxx.xx
expect {
default {exit 2}
"continue connecting yes/no?" {send "yes\n"; exp_continue}
"assword:" {send "$env(ssh_pass)\n"; exp_continue}
"sftp>"
}
send "cd /out\n"
expect {
default {exit 3}
"sftp> "
}
send "get purchase*\n"
expect "{
default exit 4
"sftp>"
}"
send "quit\n"
exit 0

And the results in my error log.

Browse : /tmp/expect.log
Record : 1 of 4 by 14 Column : 1 68 by 79
Control :

....+....1....+....2....+....3....+....4....+....5 ....+....6....+....7....+....
************Beginning of data**************
invalid command name "@{Za¤¢ a     a   a §   £@` "
while executing
"@{Za¤¢ a     a   a §   £@` "
(file "myscript.exp" line 1)
************End of Data********************

[Scott Klement]

Expect will only understand ASCII data, it sseems that your expect script was written in EBCDIC.

[RobertC]

Can I re-type the script into notepad, save it as a .txt file and then copy it into the IFS of my iSeries?

[Scott Klement]

Notepad will use the windows end-of-line convention, but... you could use it, and then use EDTF to change the end of line

Or you could take your EBCDIC file and convert it to ASCII using the CPY command.

[RobertC]

Scott,
Thanks very much for the great tips. I feel like I am close to success now. So I used the copy command to create my script in ASCII and it worked. Expect can now understand the script and it does try to establish a secure connection to the remote server. However, it says the ''authenticity of my host server can't be established.'' Here is my error log. I have x'ed out the host name, ip address and the ECDSA fingerprint information.

Browse : /tmp/expect.log
Record : 1 of 6 by 14 Column : 1 78 by 79
Control :

....+....1....+....2....+....3....+....4....+....5 ....+....6....+....7....+....
************Beginning of data**************
spawn sftp accountxxxxxxxxxxx@xxxx.xxxx.xx

The authenticity of host 'xxxx.xxxx.xx (xx.xxx.xxx.xx)' can't be established.
ECDSA key fingerprint is xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx.

Are you sure you want to continue connecting (yes/no)?
************End of Data********************



.

[Scott Klement]

If you look at the example you took from me, posted above, you'll see that it tries to find the phrase "...continue connecting..." and answer it with "yes". It doesn't appear to be working because your copy is asking the question slightly differently than the script is looking for. There are parenthesis in the question, but not in the phrase it's looking for.

That's how Expect works, it looks for strings on the screen, and responds to them.

Just update your script to match what's on the screen.

[RobertC]

I have changed my script to what's on the screen. It appears to have connected and added the ECDSA key to the list of known hosts. The Password is then prompted 3 times. It doesn't indicate if it was accepted or denied.

Browse : /tmp/expect.log
Record : 1 of 9 by 14 Column : 1 106 by 79
Control :

....+....1....+....2....+....3....+....4....+....5 ....+....6....+....7....+....
************Beginning of data**************
spawn sftp accountxxxxxxxxxxx@xxxx.xxxx.xx

Warning: Permanently added the ECDSA host key for IP address 'xx.xxx.xxx.xxx' to the list of known hosts.

Password:
Password:
Password:

accountxxxxxxxxxxx@xxxx.xxxx.xx's password:
************End of Data********************

[Scott Klement]

When it repeats the password prompt, it means it was denied. If it were accepted, it would've continued past the prompt.

[RobertC]

Scott,
I have resolved the password issue and connected to the remote server. However, the syntax is giving me problems. Do I have to use send for every simulation of an entry? I assume that to change directory I must do a {send "cd /directoryname/n" ; exp_continue} and then use "sftp>'' to setup the command line for the next instruction. Does the default {exit} set an error condition and get out? I simply want to go to the ''out' directory, get purchase orders and exit. Please lend me some insight.


Browse : /myscriptbk.exp
Record : 1 of 22 by 14 Column : 1 87 by 79
Control :

....+....1....+....2....+....3....+....4....+....5 ....+....6....+....7....+....
************Beginning of data**************
#!/usr/local/bin/expect -f
set timeout 20
spawn sftp accountxxxxxxxxxx@xxxx.xxxx.xx
expect {
default {exit 2}
"Are you sure you want to continue connecting (yes/no)?" {send "yes\n"; exp_continue}
"Password:" {send "xxxxxx\n"; exp_continue}
"sftp>"
}
expect {
default {exit 2}
{send "cd /out\n"; exp_continue}
"sftp>"
}
expect {
send "get purchase*\n ; exp_continue}
default {exit 2}
"not found'' {exit 3}
"sftp>"
}
send "quit\n"
exit 0
************End of Data********************

[RobertC]

My error log indicates ''More than one End of Line character found.'' Is there a way that I can make sure that my script is handling End of Line properly? I am going into Edit Services and changing my EOL option to *LF. Does Field Exit (Field+) replicate a line feed? I need to make sure that Expect understands where the end of each instruction line is.


Browse : /tmp/expect.log
Record : 1 of 6 by 14 Column : 1 59 by 79
Control :

....+....1....+....2....+....3....+....4....+....5 ....+....6....+....7....+....
************Beginning of data**************
spawn sftp accountxxxxxxxxxxx@xxxx.xxxx.xx

Password:
Connected to xxx.xxxx.xx...

sftp>
************End of Data********************


F3=Exit F10=Display Hex F12=Exit F15=Services F16=Repeat find
F19=Left F20=Right
More than one End of Line character found. +

[Scott Klement]

Field exit is unrelated to line feeds.

Use the F10=Display Hex option to see where the extra/wrong line feeds are, then re-do that line.

Or, use a real editor ;-)

[RobertC]

I thought I could handle EOL in EDTF command by going into F15 Services and using *LF as the EOL option,

[RobertC]

It seems to be reading the script fine until it logs in with the password. I am connected to the remote server but now my error log keeps displaying sftp>.


Browse : /tmp/expect.log
Record : 1 of 6 by 14 Column : 1 59 by 79
Control :

....+....1....+....2....+....3....+....4....+....5 ....+....6....+....7....+....
************Beginning of data**************
spawn sftp accountxxxxxxxxxx@xxxx.xxxx.xx

Password:
Connected to xxxx.xxxx.xx...

sftp>
************End of Data********************

[Scott Klement]

sftp> is the prompt where it asks you for a command, so that's good.

Your script, above has this as the next step (which doesn't make sense, I think it's a mistake):

expect {
  default {exit 2}
  {send "cd /out\n"; exp_continue}
  "sftp>"
}

The "expect" command tells it to wait for various strings to appear on the screen. However, you're doing a 'send' command without a string to wait for, and then "exp_continue" which basically tells it to repeat the same condition.

my guess is that you really wanted to do this:

send "cd /out\n";
expect {
  default {exit 2}
  "sftp>"
}

This says to send the "cd /out" (plus \n, which means "press enter") then wait for "sftp>" for up to 2 seconds, if you don't get it in 2 seconds, exit the script with error code 2. You might also want to have code in there that checks for an error like that the directory isn't found, but I don't know off the top of my head what the message for the directory not found looks like, so don't know what string to wait for.

The next section has the same problem, it should probably be something like this:

send "get purchase*\n";
expect {
  default {exit 2}
  "not found" {exit 3}
  "sftp>"
}

But, that's the general idea with expect is that you're sending a command to the screen, then waiting for a response. So send "cd /out" and wait for the "sftp>" to come back. Then send "get purchase*" and wait for the "sftp>" to come back (or "not found"), etc. Your original code was going into an 'expect' group, which is waiting for a string without sending the command first. I hope that makes more sense, now?

[RobertC]

Thanks so much for the response. It makes a lot more sense now indeed. I was really struggling with Expect syntax so getting your feedback in this forum has been invaluable. I have a much better understanding of it now.

I also changed my CL program to call my script with expect -d instead of expect -f. By calling the script with the debugger option on there was so much more information available. In my error log (expect.log) I am now seeing exactly what Expect was matching on and what it wasn't. The main problem I had is that I was looking for the string: "Are you sure you want to continue connecting (yes/no)?"

I was getting frustrated as I was sending "yes\n" but it appeared to not be accepting it. Once I put the debugger on it was easier to understand that the remote server wasn't displaying that string at all. I was trying to match something that wasn't on the screen.

It was just looking for the Password. Once I got over that hurdle the program did access the files and pulled them over to my directory on the iSeries. I have included my script below.

Thanks for all the help Scott. You are the best !


#!/usr/local/bin/expect -f
spawn sftp accountxxxxxxxxxxx@xxxx.xxxx.xx
expect {
"Password:" {send "xxxxxx\n"}
}
expect {
"sftp>"
}
send "lcd /xxxxxxxxxxx\n"
expect {
"sftp>"
}
send "cd /out\n"
expect {
"sftp>"
}
send "get Purchase*.edi\n"
expect {
default {exit 2}
"not found" {exit 3}
"sftp>"
}
send "quit\n"
exit 0