ibmi-brunch-learn

Announcement

Collapse
No announcement yet.

Large string in YAJL

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Large string in YAJL

    I'm using Scott Klement's port of YAJL to parse JSON I'm receiving in a response from a web service. It works great, except I'm having trouble with one of the objects. The service sends a PDF document as a base64-encoded string. The document is typically 450-500KB, and the maximum field size in RPG (relevant to the return value of yajl_get_string()) is 64K, so when I try to pull the base64 string out of the JSON node and write it to a temporary text file (in preparation for decoding it from base64), the target file only gets the first 64K of the data. An example of the code I've tried is:

    Code:
    yajl_save_string_stmf(pdfNode : outFile : errMsg);
    Where pdfNode is the yajl_val pointing to the node containing the long string.

    My initial research suggests I should write yajl_val_t.string to the stream file, but this is pretty new territory to me, so I'm mindful of Scott's recommendation not to use that data structure directly from RPG. Can anyone advise as to whether I'm barking up the right tree, or suggest an appropriate solution?

    Thanks!

  • #2
    Re: Large string in YAJL

    yajl_get_string() is limited to 64k for compatibility with V5R3/V5R4.

    But, are you saying that yajl_save_string_stmf() is only writing the first 64k of the string? That should not be the case... This works with pointers, so shouldn't be limited to 64k...

    Comment


    • #3
      Re: Large string in YAJL

      For some reason I assumed yajl_save_string_stmf() was calling yajl_get_string() and hitting the limitation that way. Reviewing the source, I see that assumption was clearly wrong.

      Aha. Your mention of V5R3/V5R4 compatibility got me going in what I think is the right direction. My compiler defaults are for V5R4, for compatibility with customers' boxes. Looks like there is a 64K limit to the %str() BIF in V5; that's most likely where I'm getting hung up.

      From yajl_save_string_stmf():
      Code:
      callp write( fd: nv.string: %len(%str(nv.string)) );
      Thanks as always, Scott!

      The next part of this process, of course, needs to decode the base64 text to the binary PDF file. I was originally hoping to use the DB2 SQL BASE64DECODE UDF, as it's to my advantage to use native support whenever possible, but it appears that function may be limited to 4 KB. Is there a native way to decode a large string, or is my best bet still the Klement BASE64R4 utility?

      Comment


      • #4
        Re: Large string in YAJL

        Thanks, it didn't occur to me that %str would be limiited to 64k. I will consider using another method for yajl_save_string_stmf().

        As for the SQL BASE64DECODE, I haven't used that, myself. If you want input from other people, I would recommend starting a new discussion thread, as many people who don't work with YAJL will simply skip reading this one.

        Comment


        • #5
          Re: Large string in YAJL

          Personally, I wouldn't worry about getting rid of the %str(), as V5 is now two full releases back.

          Comment


          • #6
            I think I'm hitting this limitation myself. I'm working with image files. Since I have the base64 code already, I was just going to send that from my JS front end to my RPG. Longer strings are getting cut off around the 65,536 character.

            Maybe I'll go back to using FUPLOAD and then converting the image to a blob in the RPG.
            Your friends list is empty!

            Comment


            • #7
              You think you're hitting this limitation, but not sure? Did you compile YAJL with the V5R4 RPG compiler, and are calling yajl_save_string_stmf in your program? If so, you are subject to this limitation. If not, you are not subject to this limitation.

              Please keep in mind that aupport for V5R4 was discontinued in 2013. So if you're still running it, this is likely the least of your problems.

              But, you can work around this limiation by using pointers and dynamic memory. You can either use yajl_get_string_buf or yajl_get_string_utf8 to get a pointer to the full string, and then write it yourself. There's not much I can do to make it easier than that because V5R4 character/varying fields simply do not have the ability to store larger values than 65535, so your only option is to use pointers.

              Its not clear what this has to do with FUPLOAD... if you were using FUPLOAD and it worked, why did you change to a different process that involves JSON and base64? Seems odd to make the change, unless you had some compelling reason (such as a customer who demanded it) in which case, I don't see how going back is a valid option.

              Comment


              • #8
                Fast-forward 6 years. . . Has this size limit issue been fixed in any way? It is 2020 and I have 100% the same scenario: A base64 encoded pdf, and yajl_get_string is cutting off at 64K.

                Capturing the whole response is not an issue, but parsing the pdf out of the response hits the 64K limit.

                Comment


                • #9
                  Yes, yajl_get_string() is, and always will be, limited to 64k.

                  This thread was about yajl_save_string_stmf(), which should not have been so limited -- and that was fixed at the time of this thread.

                  Comment


                  • #10
                    There is also yajl_get_string_buf() if for some reason you'd prefer not to save the data to disk. This also does not have a 64k limitation.

                    Comment

                    Working...
                    X