ibmi-brunch-learn

Announcement

Collapse
No announcement yet.

First attempt at Data-Into with YAJL parser

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #16
    I don't think there is a workaround Scott - at least not until there is a widespread implementation of a comprehensive schema system for JSON - and even then, just like with XML schemas, they would only be used 30% of the time if that.

    What I was thinking of was a tool that would simply attempt to generate the required names/hierarchy together with the DIMs etc. where needed. Not a complete solution, but for large complex documents it would sure as heck save some time!

    Comment


    • #17
      Originally posted by JonBoy View Post
      I don't think there is a workaround Scott - at least not until there is a widespread implementation of a comprehensive schema system for JSON - and even then, just like with XML schemas, they would only be used 30% of the time if that.

      What I was thinking of was a tool that would simply attempt to generate the required names/hierarchy together with the DIMs etc. where needed. Not a complete solution, but for large complex documents it would sure as heck save some time!
      Agreed! That is really what I was thinking as well... maybe just establish the DS elements & sizes based on the sample data in the JSON. (I don't see how a tool could ever get it exactly right)... but it would at least remove the issues associated with name mismatches and data structure nesting.

      It would then be up to the developer to adjust the data structure. In the case of this API, 50 is the max.

      Comment


      • #18
        Just to update the original post...

        I am able to parse the JSON document and get the elements I need using YAJL without using DATA-INTO. I used the same data structure and assumed the same JSON structure. So I'm not sure what this means (if anything).

        Based on debug, it appears that YAJLINTO is parsing the entire document without error. So there is either something wrong with my DS, or something wrong with DATA-INTO (unlikely).

        The question remaining for me is... When should I use YAJL_object_loop (read each object) versus YAJL_object_find to get the specific objects I need, within the array loop?

        Comment


        • #19
          Originally posted by gwilburn View Post
          When should I use YAJL_object_loop (read each object) versus YAJL_object_find to get the specific objects I need, within the array loop?
          Not sure there are right or wrong times... use whichever makes more sense for what you're coding at the time.

          Comment


          • #20
            I have spent some time parsing this document with YAJL... I'm getting where I need to be. But the process for me is slow and tedious because of the levels of "nesting" I have to parse through to get the data I need. In some cases it (the data) is 5 levels from the document node.

            I've opened a case with IBM to see why DATA-INTO is not populating my DS.

            In the meantime, is there an easier method with YAJL to get directly to a particular nesting level without iterating through all of the nodes?

            Example:

            Code:
            addressnode = YAJL_object_find(ordernode : 'fulfillmentStartInstructions.shippingStep.shipTo.contactAddress');
            j = 0;
            dow YAJL_object_loop( addressnode: j: key: val);
              select: 
              when key = 'addressLine1'; 
                orders(i).address1 = YAJL_get_string(val);
              when key = 'city';
                orders(i).city = YAJL_get_string(val);
              etc...
              endsl;
            enddo;
            Or is there a better way to "drill down" to the correct node?

            Comment


            • #21
              Just heard back from IBM...
              The lab can replicate the issue. It is caused by a JSON document that contains empty arrays - they are currently looking into a fix
              APAR SE69868

              Comment


              • #22
                Right now there isn't a "path" type option for YAJL_object_find(). To me, it's just as easy to do something like this:

                Code:
                node = YAJL_object_find(ordernode: 'fullfillmentStartInstructions');
                node = YAJL_object_find(node: 'shippingStep');
                node = YAJL_object_find(node: 'shipTo');
                addressnode = YAJL_object_find(node: 'contactAddress');
                You could potentially do it in one line of code, but it'd be a long line, and I'm not sure that it really saves you anything

                Code:
                addressnode = YAJL_object_find(YAJL_object_find(YAJL_object_find(YAJL_object_find(ordernode: 'fulfillmentStartInstructions'): 'shippingStep'): 'shipTo'): 'contactAddress');
                There's nothing to stop you from writing your own subprocedure that calls YAJL_object_find() in a loop if that makes it simpler.

                Comment


                • #23
                  Thanks Scott... that is kind of what I did. I used node2, node3, node4 so I could keep track of "how deep" I was. That way I could pick up objects at a previous level within the same structure.

                  I experimented with your second suggestion... your right... that got ugly really quick.

                  Comment


                  • #24
                    Originally posted by gwilburn View Post
                    Just heard back from IBM...
                    The lab can replicate the issue. It is caused by a JSON document that contains empty arrays - they are currently looking into a fix
                    APAR SE69868
                    Just got round to testing this and sure enough if you remove all the items that constitute an empty array from the file it works perfectly with allowextra working as it should and ignoring all the unwanted stuff. Hopefully they will fix the path bug at the same time.

                    Comment


                    • #25
                      IBM is estimating a fix by the end of September... Unfortunately, I can't wait that long. So I'm using YAJL to parse it - while it took me longer, it is a great learning experience for someone who hasn't worked with JSON previously.

                      Comment


                      • #26
                        Just as an FYI, as an alternative to DATA-INTO/YAJL for decoding JSON, there is also SQL function JSON_TABLE():


                        Note that JSON_TABLE() has a very different purpose - it is about extracting specific values from the JSON, whereas DATA_INTO is about extracting all values.

                        Comment


                        • #27
                          Thanks... I looked at that. I thought Data-Into would be the quickest, easiest method to get the data into an RPG program where logic could be applied before placing the data into two separate tables. While Data-Into may extract all values, I only deal with those defined in my DS.

                          If I had time, I would like to learn the JSON_TABLE function.

                          Comment


                          • #28
                            JSON_TABLE() is not about placing the values into a table. It is about presenting a selection of fields from a JSON as if they were a table, that can then be queried with SQL. Kind of like a CTE or a subselect.

                            Comment


                            • #29
                              Originally posted by Vectorspace View Post
                              JSON_TABLE() is not about placing the values into a table. It is about presenting a selection of fields from a JSON as if they were a table, that can then be queried with SQL. Kind of like a CTE or a subselect.
                              Ah.... OK. When I reviewed that article it didn't "click" for me. Sometimes the SQL syntax is difficult to get my head around.

                              Comment


                              • #30
                                Originally posted by Vectorspace View Post

                                Note that JSON_TABLE() has a very different purpose - it is about extracting specific values from the JSON, whereas DATA_INTO is about extracting all values.
                                DATA-INTO _can_ extract all values, just as Bon Anniversaire could, but by specifying allowextra=yes it is perfectly possible to extract only those portions that you want.

                                I'd be interested in seeing an example how you get JSON_TABLE() to parse the kind of mess that this particular JSON is. Lots of nested elements, nested arrays, not to mention the empty arrays that are (temporarily) the cause of this particular problem. I've seen a number of examples where JSON_TABLE() was doing relatively simple extraction but nothing for complex cases.

                                Comment

                                Working...
                                X