ibmi-brunch-learn

Announcement

Collapse
No announcement yet.

"£" sign in physical file converts to odd characters in IFS file

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • "£" sign in physical file converts to odd characters in IFS file

    We are using Scott Klement's routines for creating stream files in the IFS from RPG, but for some reason the "£" sign is being converted to what displays like "xA3" in Notepad++, and all these fields are being rejected as the stream file is XML data.

    The physical file is CSID 285 (UK standard), and the XML file should be UTF-8.

    Any thoughts on how to fix this?

    Thanks
    Poddys Rambles On

  • #2
    Well it would help if we knew what code page settings etc. you were using on the open.

    Comment


    • #3
      No magic here... if you coded it correctly, it'll work correctly.

      You didn't provide much information, but from what you did provide, you're saying a single character in EBCDIC looks like multiple incorrect characters in UTF-8. To me, that sounds like it is indeed being translated to UTF-8, but then is either being misinterpreted or further translated. For example, if you made the file UTF-8, but then tried to view it as ASCII, you'd see multiple characters where it'd show as a single character in UTF-8.

      Comment


      • #4
        Please provide us more information and we will be better equipped to help you.

        All my answers were extracted from the "Big Dummy's Guide to the As400"
        and I take no responsibility for any of them.

        www.code400.com

        Comment


        • #5
          Thanks for the help everyone.
          The output file is coded as CSID 1252. It looks like for UTF-8 it should be 1208?

          The file is created and written to using your IFS routines Scott. With RPG Free Format these have been a blessing to MANY folks like us I am sure.
          The settings for the stream file are as follows:

          Code:
                  // ************************************************************* //
                  // Copy Member.: COMIIFSD                                        //
                  // Title.......: Copy Member: IFS D-Specs                        //
                  //                                                               //
                  // D-Specs for IFS file handling (Scott Klement API's)           //
                  //                                                               //
                  // Documentation: https://www.scottklement.com/rpg/ifs_ebook/    //
                  //                                                               //
                  // 29/06/2018  Andy Jeffery                                      //
                  //  Program written - Free Format Version.                       //
                  //                                                               //
                  // ************************************************************* //
          
                  // ------------------------------------------------------------- //
                  //  IFS Prototypes...                                            //
                  // ------------------------------------------------------------- //
                  // Open IFS file
                  dcl-pr Open      int(10) extproc('open');
                    Filename       pointer value options(*string);
                    Openflags      int(10) value;
                    Mode           uns(10) value options(*nopass);
                    Codepage       uns(10) value options(*nopass);
                  End-Pr;
          
                  // Write to IFS file
                  dcl-pr Write     int(10) extproc('write');
                    FileHandle     int(10) value;
                    DataToWrite    pointer value;
                    NBytes         uns(10) value;
                  End-Pr;
          
                  // Close IFS file
                  dcl-pr Close     int(10) extproc('close');
                    FileHandle     int(10) value;
                  End-Pr;
          
                  // ------------------------------------------------------------- //
                  // IFS Variables...                                              //
                  // ------------------------------------------------------------- //
                  dcl-c seek_set    const(0);
          
                  dcl-s o_wronly    int(10) inz(2);
                  dcl-s o_creat     int(10) inz(8);
                  dcl-s o_trunc     int(10) inz(64);
                  dcl-s o_CodePage  int(10) inz(8388608);
                  dcl-s o_textdata  int(10) inz(16777216);
                  dcl-s o_rdwr      int(10) inz(4);
                  dcl-s s_irwxu     int(10) inz(448);
                  dcl-s s_iroth     int(10) inz(4);
          
                  // ------------------------------------------------------------- //
                  // Standard Variables...                                         //
                  // ------------------------------------------------------------- //
                  dcl-s asciiCodePage   uns(10) inz(819);
                  dcl-s BytesWrt        int(10);
                  dcl-s Data            varchar(32000);
                  dcl-s Filename        varchar(1000);
                  dcl-s FileDesc        int(10);
                  dcl-s ReturnInt       int(10);
                  dcl-s BytesToWrite    packed(5:0);
                  dcl-c CRLF            const(x'0d25');
                  dcl-s FileCreated     ind inz(*off);

          The code for creating the stream file is as follows:

          Code:
                   // Create stream file.
                   Filename = serverLocn +
                          %trim(codevalue) +
                          streamFileFolder +
                          streamFileName +
                          timestamp +
                          streamFileSuffix;
                   FileDesc = open(%trim(Filename)
                    : o_creat + o_wronly + o_trunc + o_codepage
                    : s_irwxu + s_iroth
                    : asciiCodePage);
                   ReturnInt = close(FileDesc);
          Hope this helps. I find it confusing where there is both a codepage and Ascii codepage, although it doesn't look as if the Ascii codepage is being used. Not sure how to correctly convert a codepage to decimal, so googling that now...
          Poddys Rambles On

          Comment


          • #6
            Following my earlier response and looking again at Scott's "IFS e-book", I modified the code to create the stream file as follows:

            Code:
                     asciiCodePage = 1208;
                     FileDesc = open(%trim(Filename)
                      : o_creat + o_wronly + o_trunc + o_codepage
                      : s_irwxu + s_iroth
                      : asciiCodePage);
                     ReturnInt = close(FileDesc);
                     FileCreated = *on;
            
                     // Open for output
                     FileDesc = open(%trim(Filename)
                      : o_textdata + o_rdwr);
            It's still being created as CCSID 1252 though according to WRKLNK:
            Code:
                                                                                    
            Object . . . . . . :   /QNTC/DTINTRA/Mansys/XML_Files/Test/Customers_To >
            
            Type . . . . . . . . . . . . . . . . . :   DSTMF                         
            
            Owner  . . . . . . . . . . . . . . . . :   QDFTOWN                       
            System object is on  . . . . . . . . . :   Remote                        
            
            Coded character set ID . . . . . . . . :   1252                          
            Hidden file  . . . . . . . . . . . . . :   No                            
            PC system file . . . . . . . . . . . . :   No                            
            Read only  . . . . . . . . . . . . . . :   No                            
            
            Need to archive (PC) . . . . . . . . . :   Yes
            When I look at the file using WRKLNK I see my output with a "£" sign:

            Code:
             ************Beginning of data**************                                          
            <?xml version="1.0" encoding="utf-8"?><customers><customer><buyer_id>£04764</buyer_id>
            When I look at the file using Notepad++ however, I see:

            Code:
            <?xml version="1.0" encoding="utf-8"?>
            <customers>
                <customer>
                    <buyer_id>ð4764</buyer_id>
                    <company_name>EXPORT ACCOUNT (VATABLE)</company_name>
                    <name_address />
            I had hoped the change I made to set asciiCodePage to 1208 would have worked.
            Poddys Rambles On

            Comment


            • #7
              Please stop plastering my name all over it (especially when you can't figure it out, haha). These APIs were designed for Unix when I was still a little boy, and are system APIs provided by IBM. You may have learned how to use them from reading my writings, but they are not my routines.

              Right now, I suspect that its keeping 1252 (Windows Latin-1 ASCII) because the file already exists and is already tagged with 1252. It is simply keeping the existing CCSID on the file.

              But, you won't be able to change it to UTF-8 while still using the old V4 version of the prototype or the old O_CODEPAGE flag. UTF-8 cannot be represented as a single code page, it is a multiple code page mixed encoding, so requires CCSID support. You will need the new prototype that supports the extra parameter, and you will need the new O_CCSID and O_TEXT_CREAT flag.

              I should use the word "new" for these things, they were released in 2002, so not new... but they're newer than the definitions you're using.
              here is my copybook http://www.scottklement.com/rpg/copy...io_h.rpgle.txt

              Comment


              • #8
                With the updated copybook, you should be able to do it with a single call to open(). I do suggest deleting the file first to ensure you don't pick up the old code page.

                Code:
                unlink(%trim(filename)); // delete file
                
                FileDesc = open( %trim(Filename)
                               : o_creat + o_rdwr + o_ccsid + o_textdata + o_text_creat
                               : s_irwxu + s_iroth
                               : 1208
                               : 0 );
                
                FileCreated = *on;

                Comment


                • #9
                  Great advice, thanks Scott, and well I guess like many others we plaster your name all over the place because few others (if any) have come up with a solution to working with stream files in RPG that works as well and does as much. I think we need to update to the latest version, that would be a good start.

                  Interestingly, the file I am creating is timestamped, so it's a new file every time. Checking the CCSID of others I have created from RPG on the IFS, these are "mostly" the correct CCSID (819 the default not 1252).
                  The difference in this file is that it's being created in a folder on a windows server, mapped from the iSeries using QNTC. I'm wondering if that is part of the problem.

                  First, I'm going to try and get the updates in place, as that doesn't hurt.
                  Poddys Rambles On

                  Comment


                  • #10
                    Well I found the culprit, now to figure out how to fix the problem!
                    My XML file needs to be created in a folder on a windows server so it can be picked up by a watcher service and imported into another system. The link to the other server is using QNTC.
                    I changed the program to create the file instead in a folder on the IFS, and the CCSID was 1208, whereas if the file is created in the QNTC folder it's CCSID is 1252.
                    I hope there is a setting somewhere that allows me to resolve this. There seem to be a number of related posts on Google at least.
                    Poddys Rambles On

                    Comment


                    • #11
                      Did you change it to use the single open (vs the two opens you originally posted) that I explained?

                      Comment


                      • Poddys
                        Poddys commented
                        Editing a comment
                        Sorry, I haven't had a chance to look at it due to another higher priority project. Might have to pause this until January unless I get time to play with it. Need to fix the problem regardless.
                    Working...
                    X