ibmi-brunch-learn

Announcement

Collapse
No announcement yet.

JSON and YAJL problem

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • JSON and YAJL problem

    Hi

    i have dowloaded the JSONWEBSRV.sqlrpgle from http://www.scottklement.com/yajl/ and customized for my needs; the first program that works with few records , works fine, instead another program that works/read a file with about 10000 records give me error; this is the error;
    CGI PROGRAM /QSYS.LIB/EUR_2VAROB.LIB/WSCLI.PGM RETURNED EXCEPTION ID CEE9901
    SEE JOBLOG FOR JOB 329645/QTMHHTTP /WEBREST

    this is the joblog:

    MCH3402 Uscita 40 02/07/15 18:49:58,103936 <ingHighUse 000064 YAJLs YAJL * STMT
    From the consumer. . . . . . . . : QTMHHTP1
    From the program. . . . . . . : stringHighUse
    To the form. . . . . . . . . : YAJL_BUF
    To the procedure. . . . . . : yajl_buf_append
    Education. . . . . . . . . : 4
    Message. . . : is tried to make reference to an object, or to one
    part of an object, that doesn't exist anymore.
    Cause. . . . . : The cause most common it is that an address memorized for a
    object is not more correct in how much the object has been cancelled or of it

    RNQ0202 Copia sender 99 02/07/15 18:49:58,104592 QRNXIEs QSYS * STMT QRNXIE QSYS * STMT
    From the consumer. . . . . . . . : QTMHHTP1
    From the form. . . . . . . . : QRNXMSG
    From the procedure. . . . . . : InqMsg
    Education. . . . . . . . . : 3
    To the form. . . . . . . . . : QRNXMSG
    To the procedure. . . . . . : InqMsg
    Education. . . . . . . . . : 3
    Message. . . : Called to YAJL_ADDNU finished with error (C G D F).
    Cause. . . . . : The procedure RPG WSCLI in the program EUR_2VAROB/WSCLI
    to the education 014500 have recalled a program or a procedure YAJL_ADDNU,
    finished with error. If the name is * N, the call was a call
    connected through the pointer of the procedure. Correction. . . : For
    further information on the cause of the error to check the recording
    jobs and to contact the person responsible of the maintenance of the programs. Choices
    possible for the answer to the message. . . . . . . . . . . . . . . : D--
    To get a formatted dump RPG. S--to Get a dump of system. G--
    The elaboration to continue to * GETIN. C--to Annul. F--to Get a dump
    formatted suit.


    Is there a data limit .. ??

    Thanks

  • #2
    Re: JSON and YAJL problem

    If i try to limit the read to 200 or 500 records, the program don't give error, but in the browser i see olny and always 70 records and the last record is not complete..

    Thanks

    Comment


    • #3
      Re: JSON and YAJL problem

      Hi

      any idea ?

      here the Dump:

      Dump traccia utente per il lavoro 330241/QTMHHTTP/WEBREST. Dimensione: 300K, riavvolto 0 volte.
      --- 07/03/2015 09:08:31 ---
      00000003:516760 ExceptionHandler_Default(), exception occurred
      00000003:516800 C238ED70C8:0064F0 L:0007 /tmp/prccell-os400-mod_as_auth.qwobj.oft-aaaoYxRqa/pc-zsrvhjob.qwobj/mfsmount/zsrvhjo
      00000003:516832 C238ED70C8:0064F0 C3F2D4F1 F6F0F1 *C2M1601.........*
      00000003:516872 C238ED70C8:006530 L:0030 /tmp/prccell-os400-mod_as_auth.qwobj.oft-aaaoYxRqa/pc-zsrvhjob.qwobj/mfsmount/zsrvhjo
      00000003:516928 C238ED70C8:006530 00000000 00000000 FFFFFFFF B824D9E0 *..............Rç*
      00000003:516968 C238ED70C8:006540 D47E0202 94FFE9E0 00000000 00100000 *M=..m.Zç........*
      00000003:517008 C238ED70C8:006550 FFFFFFFF 4422D036 00000000 00000000 *......è.........*


      could be a problem storage allocation ??

      Thanks

      Comment


      • #4
        Re: JSON and YAJL problem

        I have never run into this problem, so it's hard to say what is causing it. yajl_buf_append() increases the size of a buffer, and then copies new data on to the end, so it makes sense that it might be running out of memory -- but, why would that cause an MCH3402?

        I think in order to help you, I need to be able to reproduce the problem on my own computer. Please post your code and the JSON data you're attempting to parse so that I can try it myself.

        Comment


        • #5
          Re: JSON and YAJL problem

          Looking at the code for YAJL (this is in the original YAJL implementation, not my wrappers, and is written in C) I see this in the yajl_buf_ensure_available() function:

          Code:
              while (want >= (need - buf->used)) need <<= 1;
          What this does is double the the 'need' variable until it is larger than the 'want' variable. (The <<= operator shifts the bits in a number to the left, so essentially doubles the number.) This means that when 'want' is around 8mb, 'need' can end up being around 16mb, which is larger than IBM i can handle in a single allocation (unless you use teraspace that is.) This is likely what is causing the problem... I could change the code to be a bit more conservative on this.

          Of course, this is just a guess at this point. As I said earlier, in order to know for certain if this will help, and make sure we solve the problem, I will need to be able to reproduce the problem. Please post the code and the JSON document you are attempting to parse...

          Comment


          • #6
            Re: JSON and YAJL problem

            Hi Scott

            i sent you a mail to <<'yajl@scottklement.com'>> with source program and data base file, so you can reproduce my error .....

            i would not prefer to post here the data base file ..... if not Ok (that i have sent the mail)n tell me and i try for anoter way to let you hve pgm and file

            Thank in advance

            Comment


            • #7
              Re: JSON and YAJL problem

              Hi Scott

              We take the discussion here instead of midrange.

              I have tried gio?s program and the error he gets is not related to the buffer problem. First time the buffer runs full but the program continues but produces garbage in the buffer. Since the program runs it leaves the activation group active. If you subsequent call the program again without reclaiming the activation group you will get the MCH error.

              Since YAJL runs UTF-8 internally the EBCDIC size passed to YAJL isn?t the actual size of the YAJL buffer since UTF-8 theoretical may take up 4* storage compared with the passed EBCDIC length.

              The problem may be cause by YAJL using only 16 MB storage allocation support and not teraspace but why should it then stop around 8-10 MB?

              My guess is that YAJL reallocates its buffer based on the EBCDIC length it receives and not the actual length of the UTF-8 string it writes to the buffer. This will eventually cause the buffer to run full if the EBCDIC passes other characters than equivalents to ASCII x?00?-x?7F? that equals UTF-8 1 byte encoding.

              You may wonder why I also use YAJL since I have my jsonNode() support, but YAJL in C work 3-4 times faster than the same written in RPGLE and adds the ability to read complex JSON without first converting it to XML.

              My suggestion of a yajl_getBuf(addr:size) sub-procedure make sense since we will be able to send UTF-8 directly to the Apache output buffer without first converting the UTF-8 to EBCDIC that then is converted to UTF-8 by Apache.

              The code would be as simple as:

              Code:
              ?
              // do a YAJL JSON object
              ?
              setContent(?text/plain charset=utf-8?);  // set the content, charset will disable Apache EBCDIC conversion
              yajl_getBuf(addr:size);  // get the addr and size of YAJL generation
              storeAppend(0:addr:size); // append the UTF-8 to the output buffer
              echoToClient();  // send the result to the requester
              It will also ensure that you are able to send content-type ?application/json? that some web-services requires.

              Comment


              • #8
                Re: JSON and YAJL problem

                I think I have fixed these problems, but I do not have access to a V6R1 machine to test with... can you please try the new version?

                here is what I did:

                1) Made the allocation routine more conservative. (i.e. the loop I posted earlier). This was, indeed, the main problem. As soon as gio.cot's file was 4 bytes larger than 8mb, this tried to make a buffer larger than 16mb, and that caused the main problem -- now instead of doubling the buffer size, it just adds 64k each time, allowing the buffer to grow more gradually and so will work until the buffer gets near to 16mb (instead of failing at 8mb)

                2) Fixed a bug where memory used was not cleared if YAJL ended in error, causing it to think that the buffer was already allocated when it was not.

                3) Added teraspace support. This can be turned on/off by modifying and re-running the BUILDYAJL CL program that is included with YAJL. I have it turned on by default for now. Tested this with a 100mb JSON document, it worked nicely.

                The new version is now on my web site:


                Try it and let me know if it works.

                Comment


                • #9
                  Re: JSON and YAJL problem

                  Originally posted by Scott Klement View Post
                  I think I have fixed these problems, but I do not have access to a V6R1 machine to test with... can you please try the new version?

                  here is what I did:

                  1) Made the allocation routine more conservative. (i.e. the loop I posted earlier). This was, indeed, the main problem. As soon as gio.cot's file was 4 bytes larger than 8mb, this tried to make a buffer larger than 16mb, and that caused the main problem -- now instead of doubling the buffer size, it just adds 64k each time, allowing the buffer to grow more gradually and so will work until the buffer gets near to 16mb (instead of failing at 8mb)

                  2) Fixed a bug where memory used was not cleared if YAJL ended in error, causing it to think that the buffer was already allocated when it was not.

                  3) Added teraspace support. This can be turned on/off by modifying and re-running the BUILDYAJL CL program that is included with YAJL. I have it turned on by default for now. Tested this with a 100mb JSON document, it worked nicely.

                  The new version is now on my web site:


                  Try it and let me know if it works.

                  Hi Scott

                  thanks for your reply....

                  i have dowloaded the new version.. now i have not get error, but when i my broswer i run "http://myIBMi:10071/rest/wscli.pgm/ i don't see ALL the data ..

                  it stop at "COD_CLIENTE ": "010408" while the last record should show "COD_CLIENTE":"999999" . It shows olny 69 records, while should shows 11752 rows.

                  If you query the file that i sent you, you can check this

                  Thanks

                  Pheraps i'm wrong in something ... i checked the new Srvpgm YAJLR4 created on 05 jul 2015 at 21:12:56 and used today ..

                  Comment


                  • #10
                    Re: JSON and YAJL problem

                    Hi Scott

                    I have now tested your new version on gio?s example on V6R1 and it runs without errors and I have lopped his data section 10 times generating 145MB without any problems.

                    A big cadeau for your speedy response, you willingness to help others and your commitment to share your knowledge even while you yourself is going through a tough time.

                    I have btw also noticed a number of new sub-procedures compared with your old YAJL version downloaded in jan? 2015:

                    Code:
                    I -  export symbol(yajl_writeStdout)       
                    I -  export symbol(yajl_stdin_load_tree)   
                    I -  export symbol(yajl_addCharEx)         
                    I -  export symbol(yajl_addCharStmf)       
                    I -  export symbol(yajl_exbuf_new)         
                    I -  export symbol(yajl_exbuf_concat_ptr)  
                    I -  export symbol(yajl_exbuf_concat)      
                    I -  export symbol(yajl_exbuf_free)        
                    I -  export symbol(yajl_save_string_stmf)  
                    I -  export symbol(yajl_getBuf) // <<<< that I like ;-)           
                    I -  export symbol(yajl_tree_free_rpg)
                    I haven?t gone further into these, but tested the yajl_getBuf() ctr. yajl_saveBuf() discussion as previous described. yajl_getBuf() gives about 10-15 % performance advantage.

                    Comment


                    • #11
                      Re: JSON and YAJL problem

                      Hi Gio,

                      your problem is that you uses yajl_saveBuf() and your result field isn't big enough to hold the result.

                      Comment


                      • #12
                        Re: JSON and YAJL problem

                        Gio.Cot,

                        You are calling yajl_copyBuff() to copy the result into a variable in your program. It can only copy as much data as will fit in your variable, so even thouggh YAJL has the full document, when it copies into your program you lose part of it. You can fix this by using a larger variable, or by getting the raw buffer using the new yajl_getBuf(). Or you can try the new yajl_writeStdout() routine.

                        Henrik,

                        Thank you for your kind words. I'm not familiar with the word 'cadeau', I did a Google search, and it means 'gift', as I understand it?

                        Comment


                        • #13
                          Re: JSON and YAJL problem

                          Scott

                          cadeau is a french word that has sneaked into danish and in danish means "sign of recognition; tribute" ;-)

                          In other words it is a recognition to a person who voluntary gives a “gift” to others without having to do so.
                          Last edited by Henrik Rutzou; July 6, 2015, 08:21 AM.

                          Comment


                          • #14
                            Re: JSON and YAJL problem

                            Hi Scott


                            Sorry again .. and be patient

                            I’m trying your suggestion yajl_writeStdout() , I have this trouble

                            Status: 200
                            Content-type: application/json; charset=utf-8
                            €€€€‚ËÍÄÄÁËË‚š€ÈÊÍÁŒŽ€€€€‚ÁÊÊ_ËÅ‚š €‚‚ŒŽ€€€€‚%ÑËÈ …........
                            ..................

                            is this a CCSID problem …??

                            I have modified the QCLSR/QRPGLESRC/QSRCSRV ‘ CCSID form 37 to 280 , restored the ifs sourc (without any chenga to CCSID) e and try to compile .. but I receive this errors
                            :
                            21200 - CRTCMOD MODULE(YAJL/yajl_par)
                            SRCSTMF ('/ yajl/working/src/yajl_parser.c') OPTIMIZE(40) DBGVIEW(*NONE)
                            TGTRLS(V5R4M0) TERASPACE(*YES * TSIFC) INCDIR ('/ yajl/working/src'
                            '/ yajl/include') TGTCCSID(280)
                            / yajl/working/src/yajl_parser.c, 49.20: CZM0045(30) identification
                            Non declared LLONG_MAX.
                            / yajl/working/src/yajl_parser.c, 51.44: CZM0045(30) identification
                            Non declared LLONG_MIN.
                            / yajl/working/src/yajl_parser.c, 333.39: CZM0045(30) identification
                            Non declared LLONG_MIN.
                            / yajl/working/src/yajl_parser.c, 333.57: CZM0045(30) identification
                            Non declared LLONG_MAX.
                            The form YAJL_PAR has not been created because they are verified of the
                            errors of education.
                            Compilation not result.
                            Error of function. CZM0613 not monitorato from BUILDYAJL to the education to
                            high-level 0000021200, education X'0000.'
                            CZM0613 received by the procedure BUILDYAJL. (C D THE RS)
                            CZM0613 received by the procedure BUILDYAJL. (C D THE RS )

                            Do you have idea where am I wrong



                            other question for Henrik: how can do what you said : "I have lopped his data section 10 times generating 145MB"


                            thanks in advance for your time and your patience

                            Gio

                            Comment


                            • #15
                              Re: JSON and YAJL problem

                              Gio

                              Scott?s YAJL has no CCSID problems, you can just install it on any CSSID and it runs as is in the download.

                              CCSID problems only occur when the RPGLE source code has special characters that may differ in hex values from one CCSID to another and therefore must be compiled into the objects.

                              To replicate your JSON object into a big (145MB) object I just did a ?for i = 1 to 10;? loop around your code that generates JSON rows/objects.

                              I can?t help you with yajl_writeStdout(); but I can help you running YAJL with powerEXT Core were I will make a new download tomorrow including Scott?s latest YAJL version, that is if you promise to run the installation procedure as stated ;-)

                              But please don?t try to shop between understanding/making Scotts example to work with a fall back to powerEXT (that works with the new yajl_getBuf) ? you will be wasting one of our time ;-)

                              Comment

                              Working...
                              X