ibmi-brunch-learn

Announcement

Collapse
No announcement yet.

First attempt at Data-Into with YAJL parser

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • First attempt at Data-Into with YAJL parser

    So I have read Scott's blog on using this new feature (https://www.common.org/scotts-iland-...son-data-into/) . I installed/applied the PTFs recommended and compiled a simple program see how things work with a JSON document. I set the environment variable as suggested for debug. I can see the document parsing, but my data structure is empty. Scott specifically said that problems can usually be attributed to programmers variables not matching the JSON... but for the life of me I can't see it.

    If I remove the allowmissing=yes from the options, I get an error that "the document for data-into does not match the RPG variable". I'm nearly certain that I'm doing something stupid.

    FWIW - I have tried adding "path=orders" to the OPTIONS

    Here is my simple code:
    Code:
           dcl-c  OPTIONS const('doc=file case=any allowextra=yes +
                                 allowmissing=yes');
    
           dcl-ds Orders                   qualified dim(50);
             orderId                       char(40);
             legacyOrderId                 char(40);
             creationDate                  char(25);
             lastModifiedDate              char(25);
             orderFulfillmentStatus        char(10);
             orderPaymentStatus            char(10);
           //  sellerId                      char(20);
           //  pricingSummary                likeds(subtotal_t);
           //  lineItems                     likeds(lineItems_t) dim(10);
           end-ds;
    
    
             Stmf = '/ebay_163.json';
             Data-Into Orders %data(stmf:OPTIONS) %parser('YAJLINTO');
    Here is a snippet of the beginning of the JSON document

    Code:
    {
        "href": "https://api.sandbox.ebay.com/sell/fulfillment/v1/order?limit=50&offset=0",
        "total": 2,
        "limit": 50,
        "offset": 0,
        "orders": [{
            "orderId": "110350130580-28978360001!949987000",
            "legacyOrderId": "110350130580-28978360001",
            "creationDate": "2018-08-06T14:28:50.000Z",
            "lastModifiedDate": "2018-08-06T14:29:45.000Z",
            "orderFulfillmentStatus": "NOT_STARTED",
            "orderPaymentStatus": "PAID",
    .....

  • #2
    OK... So I got it MOSTLY working by changing my DS as follows... I thought using the "path=" would eliminate having to nest yet another data strucuture

    Code:
           dcl-ds json                     qualified;
             href                          varchar(256);
             total                         char(5);
             limit                         char(5);
             offset                        char(5);
             orders                        likeds(orders_t) dim(50);
           end-ds;
    
           dcl-ds orders_t                 qualified template;
             orderId                       char(40);
             legacyOrderId                 char(40);
             creationDate                  char(25);
             lastModifiedDate              char(25);
             orderFulfillmentStatus        char(10);
             orderPaymentStatus            char(10);
           //  sellerId                      char(20);
           //  pricingSummary                likeds(subtotal_t);
           //  lineItems                     likeds(lineItems_t) dim(10);
           end-ds;
    I think my issue is the nesting... the parser completed, but when i removed the commented elements... it did not populate the lineItems (or anything after). It also did not populate the second "order"

    So Is there an easier way to do these nested data structures? This is an eBay order where the "nesting" is 6 or more deep.
    Last edited by gwilburn; August 9, 2018, 01:43 PM.

    Comment


    • #3
      OK - first off don't ever use "missing" unless you have to - and you never have to. Use countprefix and a count field instead as below.

      The problem I think is that you have coded the DATA-INTO as if Orders was the top level item - it isn't. The correct fix (if you don't want the extra data) would be to use a path= option to push the start point down the tree, but I know IBM has a problem in this area right now so probably won't work. Adding path=orders might work but if not ...

      Change the DS to something like this:

      Code:
       dcl-ds OrderData;  
          total        int(5);
          limit        int(5);
          offset       int(5);  
          count_Orders int(5);  
          dcl-ds Orders                   qualified dim(50);
             orderId                       char(40);
             legacyOrderId                 char(40);
             creationDate                  char(25);
          etc.    
      
       Data-Into OrderData %data(stmf: 'doc=file countprefix=count_ case=any allowextra=yes')  
           %parser('YAJLINTO');
      Once the parse is complete the value in count_Orders will reflect the number of elements loaded. Add similar count_ fields for any other optional elements and test the count against zero to determine if they were present or not. Don't use "missing" - it is a really bad idea.

      Comment


      • #4
        I was updating my last post when you posted this. The "path=" definitely does not work.

        I tried the countprefix... it came up with 16448 (there are 2 orders). The DS is empty after "pricingSummary"

        I'll keep plugging away.
        Last edited by gwilburn; August 9, 2018, 01:52 PM.

        Comment


        • #5
          What's the error message - it usually points at the element that is the problem?

          As to 16448 that's just because no value was placed there due to the error and you don't have any INZ on the count field or the DS as a whole. So 16,448 is just the value that blanks in the field results in.

          Did you add a count_ for the line items array? You need it there as well as for orders.

          Comment


          • #6
            No error message... I will add a count to the line items array. In my DS, I have omitted several "structures" in the JSON that I don't want/need. I can't believe that would be a problem.
            Is there a PSDS that I can check for errors?

            One thing about your code... You need to have the parent DS qualified, and the nested DS are "automatically" qualified. Not sure if this is specific to v7r3 or not.

            Comment


            • #7
              Having extra stuff is accommodated by allowextra=yes so no - it should not be a problem. But EVERY array (or optional element) needs a count otherwise (without using the evil allowmissing=yes) there will be an error.

              Did you change the target for the DATA-INTO? to be the top level DS?

              If you actually post a full set of the JSON I can "play" with it. Otherwise I'm just guessing.

              I couldn't recall if QUALIFIED was only needed if the nesting was via LikeDS and had no time to check. I don't use LikeDS for nesting any more since it is so much nicer to do it directly.

              Comment


              • #8
                Here is the JSON... i appreciate your help. I have no desire to account for all eBay's stupid arrays... Most of the data I do not need.

                I'm guessing that may be my problem. However, I saw the same issue using the allowmissing=yes. I would assume that allowextra=yes would ignore extra elements and such.
                Attached Files

                Comment


                • #9
                  Ummm - which bits DO you need?

                  Comment


                  • #10
                    Well I have the code running but have the same issue as you - only the total, limit and offset values are retrieved. I've looked at the log and can't see anything wrong. Will dig some more and let you know if I find anything.

                    Comment


                    • #11
                      I have it populating the pricingSummary... the lineItems won't populate. I don't know if the issue is with the lineItems or because I skipped a few structures before that.

                      I really just need the order number, amounts, shipping method, ship to, and line item data (item quantity and price). Since the order will be paid online before I pick it up, I need only get fulfillment data.

                      Wouldn't it be cool if someone (smarter than me) created a tool that would create the RPG data structure source code for a given JSON document.
                      Last edited by gwilburn; August 10, 2018, 07:27 AM.

                      Comment


                      • #12
                        As long as you have allowextra=yes then additional elements will not cause an issue _but_ you must have any intermediate levels coded. So if y is an array and x another array within y you cannot just skip y becauae you don't need it as such. The hierarchy must be there. If I have time later today (I'm travelling) I'll try and put together a complete DS for the JSON you sent.

                        You haven't said what you did to fix the earlier problem. Be good to post that for folks who come later. I seem to be missing a PFT on my system because what I have coded should be working. I asked Barbara Morris to check it out and it works for her - so now to find what PTF(s) I'm missing.

                        As to the DS generation - it has been on my bucket list for a while for both XML and JSON. Just haven't fond the time yet.

                        Comment


                        • #13
                          Thanks Jon...

                          To fix my original error - where the document didn't match the RPG variable - I had to add a DS (I called it json) around the Orders DS because of the hierarchy you just mentioned. Compare the DS in post #1 with post #2.

                          Originally, I had thought the path=orders would take the parser directly to that level. That did not work. So adding the data structure fixed that issue.

                          Meanwhile... I'm going to experiment parsing w/out DATA-INTO.

                          Comment


                          • #14
                            OK - so whatever PTF I am missing you must have then. I'll keep looking.

                            Comment


                          • #15
                            Originally posted by gwilburn View Post
                            Wouldn't it be cool if someone (smarter than me) created a tool that would create the RPG data structure source code for a given JSON document.
                            There's one problem with that... JSON documents don't tell you the maximum size of a string, the maximum number of elements in an array, or the maximum length/decpos of a numeric variable. With that in mind, how would this tool work?

                            I suppose it could guess (based on the data provided in a sample document) how big a string would be... for example, if your string was "NOT_STARTED" it could assume the maximum length is 11 characters... but it'd have no way of knowing if some future document might allow a longer string. You might think "why not make all strings the maximum possible length just in case?" But, if it did that, all strings would be 16 mb, so even a small JSON document would occupy a ridiculously huge amount of memory.

                            If anyone has any good ideas of how to get around that problem, I'd be willing to work on such a tool.

                            Comment

                            Working...
                            X