STRQSH and sed, tr commands

[zukes1966]

I tried to use SED and TR to remove unwanted characters form file existing on IFS using STRQSH.

I've found that sed doesn't work for regular expressions and for

Examples
1.

echo 'AB' | sed 's/[AB]/?/g'
AB

Result should be equal ='??'

but

echo 'AB' | sed 's/A/?/g'
?B

It is OK.

2.
SED change CRLF to LF if we use code like

sed -C 1250  's/[^[[:alnum:]\r\n]]/?/g' source.txt > result.txt

if in source we have CRLF in result we found LF
I could not find solution for such behaviour.

3.
SED ignore characters like xA0 in source file.
Even if we use

sed -C 1250  's/[^[[:alnum:]\r\n]]/?/g' source.txt > result.txt

where source includes x'A0' in result we still have x'A0'


For SED from PHASE it is a little be better (pkt 1 works, 2,3 does not)

For TR similar story.

It seems be dissaster.

Do you have any idea how I should solve above problems?

[Scott Klement]

On my system in QShell:



On my system with PASE (I assume you meant PASE rather than PHASE)



This all looks correct to me.

[zukes1966]

Of course PASE.

I can't see the pictures attached to the answer: "Invalid File Specified"

[Scott Klement]

I don't know why you wouldn't be able to see the pictures. I can see them.

The first one is a screenshot of a QShell session, it shows this:

 > echo 'AB' | sed 's/[AB]/?/g'
   ??

As you can see, it is resulting in ??. The second is a PASE session, showing the exact same results.

I suspect you do not have your CCSID configured properly, and this is causing the [ and ] characters to be misinterpreted.

[zukes1966]

Sed for PASE works OK but for QSHELL not.
A different path and most likely different code is used for the PASE environment.
What can misconfigured CCSID mean?
Is it by any chance that the SED only operates for CCSID= 37?

[Vectorspace]

In your 5250 emulator (the program giving you the greenscreen terminal access to the iSeries), in its configuration, you can specify a CCSID. What CCSID is it set to, and what CCSID do your iSeries jobs/tables usually have?

[zukes1966]

My 5250 session job data:

[FONT=Calibri][SIZE=2]Device recovery action  . . . . . . . . . . . . . :   *ENDJOBNOLIST        [/SIZE][/FONT]
[FONT=Calibri][SIZE=2]Time slice end pool . . . . . . . . . . . . . . . :   *BASE                [/SIZE][/FONT]
[FONT=Calibri][SIZE=2]Print key format  . . . . . . . . . . . . . . . . :   *PRTHDR              [/SIZE][/FONT]
[FONT=Calibri][SIZE=2]Sort sequence . . . . . . . . . . . . . . . . . . :   *HEX                 [/SIZE][/FONT]
[FONT=Calibri][SIZE=2]  Library . . . . . . . . . . . . . . . . . . . . :                        [/SIZE][/FONT]
[B][FONT=Calibri][SIZE=2]Language identifier . . . . . . . . . . . . . . . :   PLK                  [/SIZE][/FONT]
[FONT=Calibri][SIZE=2]Country or region identifier  . . . . . . . . . . :   PL                   [/SIZE][/FONT]
[FONT=Calibri][SIZE=2]Coded character set identifier  . . . . . . . . . :   870                  [/SIZE][/FONT]
[FONT=Calibri][SIZE=2]Default coded character set identifier  . . . . . :   870                  [/SIZE][/FONT][/B]
[FONT=Calibri][SIZE=2]Character identifier control  . . . . . . . . . . :   *DEVD                [/SIZE][/FONT]
[FONT=Calibri][SIZE=2]Job message queue maximum size  . . . . . . . . . :   64                   [/SIZE][/FONT]
[FONT=Calibri][SIZE=2]Job message queue full action . . . . . . . . . . :   *PRTWRAP             [/SIZE][/FONT]
[FONT=Calibri][SIZE=2]Allow multiple threads  . . . . . . . . . . . . . :   *NO                  [/SIZE][/FONT]
[FONT=Calibri][SIZE=2]Auxiliary storage pool group  . . . . . . . . . . :   *NONE                [/SIZE][/FONT]
[FONT=Calibri][SIZE=2]Spooled file action . . . . . . . . . . . . . . . :   *KEEP    [/SIZE][/FONT]

After QSH

echo $QIBM_CCSID
0

DSPSYSVAL

[FONT=Calibri][SIZE=2][COLOR=#1F497D]                             Display System Value                  [/COLOR][/SIZE][/FONT]
[FONT=Calibri][SIZE=2][COLOR=#1F497D]System value . . . . . :   QCCSID                                  [/COLOR][/SIZE][/FONT]
[FONT=Calibri][SIZE=2][COLOR=#1F497D]Description  . . . . . :   Coded character set identifier          [/COLOR][/SIZE][/FONT]
[FONT=Calibri][SIZE=2][COLOR=#1F497D]Coded character set                                                [/COLOR][/SIZE][/FONT]
[FONT=Calibri][SIZE=2][COLOR=#1F497D]  identifier . . . . . :   870        1-65535         [/COLOR][/SIZE][/FONT]
[FONT=Calibri][SIZE=2][/SIZE][/FONT]

[Scott Klement]

https://www.ibm.com/support/knowledgecenter/en/ssw_ibm_i_74/rzahz/rzahznls.htm

[zukes1966]

After setting ENVVAR LANG = '/ QSYS.LIB / PL_PL.LOCALE' SED under QSH works correctly.
Scott thanks a lot

[zukes1966]

So, maybe someone use my result.
Below you can find a QSH command which replaces unwanted characters with spaces for data.txt file.

Result file is:

• Windows CRLF text file
• has the same CCSID as source file

Characters outside the set
"a-zA-Z0-9śŚćĆńŃęĘąĄłŁżŻźŹ|,._ -()\+"
are unwanted characters

cat data.txt | sed  's/[^a-zA-Z0-9śŚćĆńŃęĘąĄłŁżŻźŹ|,._ -()\+]/ /g' | awk '{printf "%s\r\n", $0}'   >   temp.out  && cp -p -t temp.out data.txt

Or if you use the same code for different files with different CCSID

rm -f temp.out
cat data.txt | sed  's/[^a-zA-Z0-9śŚćĆńŃęĘąĄłŁżŻźŹ|,._ -()\+]/ /g' | awk '{printf "%s\r\n", $0}'   >   temp.out  && cp -p -t temp.out data.txt