ibmi-brunch-learn

Announcement

Collapse
No announcement yet.

Implementing Rest service with Json String and Scott's yajl library

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Implementing Rest service with Json String and Scott's yajl library

    Hi,

    I am developing a program using Scott's yajl libraries to process the Json String into DB2 table. I have two questions,

    1. More generic, we are using V 7.2 - Is it ideal to use yajl libraries or we need to consider using DATA-INTO and Json tables for efficiency. Json data is expected to be passed as a string to the program

    2. I am going with the approach of Yajl and stuck with yajl_array_loops when processing the nested data structures. couldn't find any example online. also, the eval-corr is throwing error when i map the nested data structure as operand is not valid. using the cust program as example!

    Example:

    OrderData
    Orderlines(Array)
    Orderlinesinfo(Array)

    thanks for all the help as always!












  • #2
    DATA-INTO does nothing by itself, it has to be used with a "parser" program. YAJL provides a parser program named YAJLINTO that can be used with DATA-INTO to process JSON.

    You can also choose to use YAJL directly by calling its subprocedures.

    It does not matter which you use -- from an efficiency perspective, they are the same. Many people find DATA-INTO easier to code and read. There are things that YAJL can do by itself that cannot be done with DATA-INTO, for example, reading elements when their names aren't known at compile time.

    I don't understand your question about YAJL_ARRAY_LOOP.

    Comment


    • #3
      thanks for the insight.

      here is the sample piece of code,


      dcl-s docnode like(yajl_val);
      dcl-s data like(yajl_val);
      dcl-s dataNode like(yajl_val);
      dcl-s line like(yajl_val);
      dcl-s lineNode like(yajl_val);
      dcl-s coupon like(yajl_val);
      dcl-s couponNode like(yajl_val);
      dcl-s val like(yajl_val);
      dcl-s i int(10);
      dcl-s j int(10);
      dcl-s k int(10);
      dcl-s field varchar(50);

      //processing JSON data

      i = 0;
      dow yajl_object_loop(data: i: field: dataNode);
      select;
      when field = 'productYear';
      order.productYear = yajl_get_number(dataNode);

      // i have an DS array as a field so coded as below

      when field = 'lineItems';
      line = yajl_object_find(dataNode: 'lineItems');
      j = 0;
      // something wrong in below part

      dow yajl_array_loop(line:j:linenode);
      val = yajl_object_find(linenode:'productYear');
      order.lineItems(j).productYear = yajl_get_number(val);

      // second ds array as field

      val = yajl_object_find(lineNode:'coupons');

      k = 0;
      dow yajl_array_loop(val: k:couponNode);
      val = yajl_object_find(couponNode:'code');
      order.lineItems(j).coupons(k).code
      = yajl_get_String(val);



      Regards,
      Vinoth







      Comment


      • #4
        Just create your DS to match the JSON structure and use YAJLINTO. Save yourself the headache of the code above.

        Comment


        • #5
          thanks for your suggestion, I am trying parser due to time constraint - I am facing challenge with array dimensions, the parser fails with reason code 2 when processing the DS array within a DS array.. This might be basic - but, not sure how to handle this in parser.


          I have an Object which has a DS array, which itself has a DS array.

          Object
          {
          DSARRAY1 [
          {
          DSARRAY2[
          {
          }
          ]
          }
          ]
          }

          I defined DSARRAY1 AS DIM(999) and DSARRAY2 as DIM(3). But, my json data has record for only one array element for array1 and array2. the parser failed to process, I changed the dimension to 1 in both arrays and it worked.

          So, can you tell me what options I need to add in DATA-INTO, so it determines the number of elements to process the file. I used below statement,


          DATA-INTO order
          %DATA(jsonfile:'doc=file case=convert countprefix=num_')
          %PARSER('YAJLINTO');

          Regards,
          Vinoth















          Comment


          • #6
            Your DS element names must match the JSON exactly (case is not important given the options your using). Scott added a utility called YAJLGEN that will read a JSON document and create the necessary data structures in a source member. Saves a TON of work.

            You will need to adjust the size of each element and adjust the array dimensions to handle "maximum" values you might encounter.

            Most REST services will allow you to specify how many "orders" (for example) you receive in each request - most will have a maximum.

            One of the REST APIs i work with has a maximum of 100 orders, but you can request them in "pages" of 10 to 100.

            Hope that helps

            Comment


            • #7
              As an experiment change the options to include allowmissing=yes - If that runs to completion then it means that your countprefix related fields were not specified correctly - most likely they are in the wrong place ion the DSS - but since you didn't include either the JSON or the DS you tried to use I can't be any more definitive.

              Comment


              • #8
                Vinoth,

                I will repeat the code you posted above, because I found it very hard to read. Here it is, reformatted:

                Code:
                dow yajl_object_loop(data: i: field: dataNode);
                
                   select;
                   when field = 'productYear';
                      order.productYear = yajl_get_number(dataNode);
                
                   // i have an DS array as a field so coded as below
                
                   when field = 'lineItems';
                
                      line = yajl_object_find(dataNode: 'lineItems');
                
                      j = 0;
                
                      // something wrong in below part
                
                      dow yajl_array_loop(line:j:linenode);
                         val = yajl_object_find(linenode:'productYear');
                         order.lineItems(j).productYear = yajl_get_number(val);
                
                         // second ds array as field
                
                         val = yajl_object_find(lineNode:'coupons');
                
                         k = 0;
                         dow yajl_array_loop(val: k:couponNode);
                            val = yajl_object_find(couponNode:'code');
                            order.lineItems(j).coupons(k).code = yajl_get_String(val);
                I haven't seen your document, so I don't know if this code is right or wrong, but it does seem a little strange. From your code, it appears that the document is meant to look like this:

                Code:
                {
                  "productYear": 2019,
                  "lineItems": {
                     "lineItems": [
                        { "productYear": 1234, "coupons": [ "string", "string", "string"] }  
                     ]
                  }
                }
                Is that right?

                It seems strange to me that 'lineItems' would contain another 'lineItems' like that, but this is what you have coded, here:

                Code:
                  when field = 'lineItems';
                
                      line = yajl_object_find(dataNode: 'lineItems');
                As you can see... you already have a field named 'lineItems' and you're looking for another field named 'lineItems' within it.

                Then you have this:

                Code:
                      dow yajl_array_loop(line:j:linenode);
                         val = yajl_object_find(linenode:'productYear');
                         order.lineItems(j).productYear = yajl_get_number(val);
                
                         // second ds array as field
                
                         val = yajl_object_find(lineNode:'coupons');
                
                         k = 0;
                         dow yajl_array_loop(val: k:couponNode);
                            val = yajl_object_find(couponNode:'code');
                            order.lineItems(j).coupons(k).code = yajl_get_String(val);
                This says that the 2nd 'lineItems' discussed above is an array, and each element in that array is an object containing fields named "productYear" and "coupons". productYear is a number, and coupons is yet another array that contains strings.

                Is that correct?

                If so, it looks to me like the part of the code that you provided should work. But, I'm guessing that this isn't really what the document looks like... and that you've coded it incorrectly.

                Comment


                • #9
                  @Scott. thank you.

                  yes, my code is wrong in this case- the document has one lineItem only. I coded it this way with wrong assumption, the coupons part is working correctly. Now I have updated the lineItems part and it is working correctly as well - thanks.

                  my document looks like below

                  Code:
                  {
                    "productYear": 2019,
                    "lineItems": [
                      {
                        "productYear": 2019,
                        "coupons": [
                          {
                            "code": "string",
                            "percent": 0,
                            "amount": 0
                          }
                        ]
                      }
                    ]
                  }

                  I am trying load this information from DS to DB2 table using eval-corr. can you advise whether eval-corr not suitable for multi-level data structure, the program does not accept below statement.

                  Code:
                   dcl-ds icoupons likeds(coupons_t) dim(3);
                   eval-corr icoupons = order.lineItems.coupons;
                  Thanks for your time

                  Regards,
                  Vinoth

                  Comment


                  • #10
                    Excuse my ignorance. I made changes to the program and properly used eval-corr statement in a loop to process the data-structure array. program working as expected now.

                    Now, I have to just add some exception procedures and comments to beautify my code.

                    thanks everyone for the help! please comment for any suggestions and references for best code practice.

                    Comment


                    • #11
                      Hi,

                      Sorry - I have a silly question.

                      I managed to process the JsonData using the YAJL library- However, i have a specific requirement to pass the JSONINPUT as a string parameter to the program.

                      I have defined the input parm as below, but the jsonString is getting corrupted during the program call everytime. My program has multiple return statement, so I added an exit proc to clear the variable on return - still when the program is called from command line, the empty spaces in the variable takes junk characters.

                      How can i avoid this.

                      Code:
                             dcl-pi CreateJson;
                               jsonString varchar(99999);
                               success ind;
                               errorMsg char(500);
                             end-pi;


                      Regards,
                      Vinoth





                      Comment


                      • #12
                        After some research. I understand the problem is in the usage of varying length field in the parameter is causing the problem. When I tried to test the program as service, i am getting below error in the logs.

                        RNX0115 : Length of varying length variable is out of range

                        Details message:

                        Message ID . . . . . . : RNX0115 Severity . . . . . . . : 50
                        Message type . . . . . : Escape
                        Date sent . . . . . . : 10/22/19 Time sent . . . . . . : 09:56:15

                        Message . . . . : Length of varying length variable is out of range.
                        Cause . . . . . : The length of a varying length character or DBCS variable
                        is less than 0 or greater than its declared maximum length in RPG procedure

                        Could someone share an example on usage of varying length field in the program input and how to call the program externally.

                        Comment


                        • #13
                          Show us the code that is making the call.

                          Comment


                          • #14

                            Call is initiated from TIBCO IBM adaptor! Did some research online and per my understanding when the length is over 65535 - I need to provide the byte definition as 4. The default is 2 byte for varchar which is causing the program to fail and corrupt the input data. The issue is now resolved with below change.
                            Code:
                               
                                    dcl-pi CreateJson;          jsonString varchar(99999:4);          success ind;          errorMsg char(500);        end-pi;

                            Comment


                            • #15
                              The default is only 2 bytes for strings that are 65535 or smaller. The default is 4 for anything larger.

                              Comment

                              Working...
                              X