JSON File Generation

Change log:

Description

The purpose of this document is to provide a description of the JSON file export feature of the File Processor framework. 

1 - JSON Generation

To be able to export data from Rator database tables into a JSON file, some generic utilities were created in Rator File Processor framework.

This page describes the design of the framework and also some examples to use it in other parts of the system.

2 - Overview

The generation of a JSON-file is done according to a database configuration where it is possible to configure all the tags that must be printed, their structure and how data can be retrieved to fill their content.

Once the configuration is ready the JSON-producer just needs the data that will be passed as a map.

{"voucher_file" : {"currency" : "XYZ","amount" : "200",
"vouchers" : {
"voucher" : {"id" : "11111","serial_number" : "11111","secret_code" : "11111"}
"voucher" : {"id" : "22222","serial_number" : "22222","secret_code" : "22222"}
"voucher" : {"id" : "33333","serial_number" : "33333","secret_code" : "33333"}
"voucher" : {"id" : "44444","serial_number" : "44444","secret_code" : "44444"}
"voucher" : {"id" : "55555","serial_number" : "55555","secret_code" : "55555"}
}
}
}

In this example the map can contain 2 objects: “HEADER” will contain the billing source object used to fill the "currency" and "amount" key/value pairs. The keys and the methods to retrieve the data from the map and from the objects inside the map will be specified in the configuration table called JSON_TAG_DEF.

2.1 - Encoding

The encoding used to generate the file is UTF-8..

3 - Design

The persistent objects introduced in this framework as part of the configuration are JSONTagDef (table JSON_TAG_DEF) and DocumentDef (table DOCUMENT_DEF). The configuration also uses the ValueFormatter (table VALUE_FORMATTER).

The valueFormatter can also be used outside the JSON-generation.

3.1 - Data model

3.1.1 -  JSON File model

The following tables are presented in the data model diagram below:

1. CDR_FILE_DEF (class CdrFileDef)
2. CDR_RECORD_DEF (class CdrRecordDef)
3. DOCUMENT_DEF(class DocumentDef)
4. JSON_TAG_DEF (class JSONTagDef)
5. VALUE_FORMATTER (class ValueFormatter)
   

All the above mentioned classes are persistent.

3.1.2 - File Processor Model

3.2 - JSONTagDef

A record in the JSON_TAG_DEF defines a single tag (key/value pair) of the resulting JSON file, and it has the following attributes:

3.2.1 - Columns

3.3 - DocumentDef

An DocumentDef models the definition for a document type for example JSON. Here a filler class can be defined that will contain the logic to fill the map with the data, the methods to be called and also the reload method.

3.3.1 - Columns

3.4 - DocumentFiller

An abstract class DocumentFiller has been defined with an abstract method populate that will generate the dataBeans map according to a context map.

The concrete sub-classes of DocumentFiller class should contain the specific logic for the generation of the document.

Following Concrete sub-classes of DocumentFiller can be found in the FileProcessor Integration project. See JSONDocumentFiller for more details

3.4.1 - JSONDocumentFiller 

The JSONDocumentFiller class contains the logic for generating the JSON-files from data obtained by joining some database tables by executing the SQL-script configured via the corresponding DOCUMENT_DEF record.  If the desired JSON-file format cannot be achieved with JSONDocumentFiller sub class, A new subclass of  DocumentFiller must be created as per the requirements.

Instructions to set up file processor configuration for JSON-file generation with JSONDocumentFiller :

• Create JSON_TAG_DEF entries for the required JSON file format: A record in the JSON_TAG_DEF defines a single element (key/value pair) of the resulting JSON file. It is possible to define parent tag for the JSON_TAG_DEF entry, and thus it is possible to define the full hierarchy of the JSON elements. Please choose proper FORMATTER_ID (ID of the VALUE_FORMATTER table record, used to format the value before writing it to the JSON file) for each JSON_TAG_DEF entry based on the value type.
• Create a DOCUMENT_DEF record with FILLER_CLASS = 'com.CDRator.billing.document.json.JSONDocumentFiller', ROOT_TAG_ID = '<Id of the JSON Root Tag entry from JSON_TAG_DEF entries created for this JSON file format>' 
• The SQL_LOAD_RECORDS column of the above created DOCUMENT_DEF record must contain the SQL query to obtain the records that should be exported to the file. This SQL string can contain the label <BILLING_SOURCE_ID> which will be replaced with appropriate value at run-time.
  If this SQL query string contains the label <BILLING_SOURCE_ID>, it will be replaced with the ID of the current source record currently being processed to create the JSON file. Note: The billing source table name is configured in the column BILLING_SOURCE_TABLE_NAME of the table CDR_FILE_DEF. <BILLING_SOURCE_ID> will be replaced with the ID of the current row being processed from the table configured in the column CDR_FILE_DEF.BILLING_SOURCE_TABLE_NAME.
• The SQL_RELOAD_RECORDS column of the above created DOCUMENT_DEF record must contain the SQL query to load the next batch of records with ID greater than the last loaded maximum ID, which are to be exported into the JSON file. This SQL query can contain the tags <BILLING_SOURCE_ID> and <MAX_LOADED_ID> labels which will be replaced with appropriate values at run-time
• Create a CDR_FILE_DEF record with CODE ='JSON' , BILLING_SOURCE_TABLE_NAME =<Source table name>, BILLING_SOURCE_CLASS= <Java class for Billing Source>, EXPORT_TYPE = '<The Export_Type for the Customer this JSON file is generated for>', DOCUMENT_DEF_ID = '<ID of the DOCUMENT_DEF record created in above step>', FILE_NAME_SUFFIX = 'json'. Please note that the CDR_PARSER_ID, RECORD_SEPARATOR_ID are not used for JSON file export and hence they can be left empty.
• A post processor method name, post processor hookpoint key, filename format and/or filename generator method can be configured on this CDR_FILE_DEF as needed.
• Create a CDR_RECORD_DEF record with CODE ='DETAIL' , RECORD_TABLE_NAME = <DB table name for the records to be exported>, RECORD_CLASS = <Java class name for the Records to be exported>, CDR_FILE_DEF_ID = '<ID of the CDR_FILE_DEF created in above step>'. The JSON file processing engine will only need a CDR_RECORD_DEF entry for DETAIL records. So HEADER, TRAILER record definitions are not needed.
• Create a CDR_FILE_PROCESSOR entry with CODE='GENERATOR',  CDR_FILE_DEF_ID = '<ID of the CDR_FILE_DEF created in earlier step>', FILE_TRANSMITTER_ID= '<Id of the corresponding FILE_TRANSMITTER entry>' and BRAND_ID = '<Corresponding brandId>' 

Important Note: It is mandatory to configure both DOCUMENT_DEF.SQL_LOAD_RECORDS and  DOCUMENT_DEF.SQL_RELOAD_RECORDS columns with appropriate SQL queries, because the File processor JSON filler class is configured to load only a maximum 10000 records in one select. If there are more records in that Voucher Batch, the reload_method  column of the relevant JSON_TAG_DEF entry must be set to 'reloadRecords'. In that case, after printing the first set of 10000 records, the JSONDocumentFiller tries to load next batch of records with IDs greater than the ID of the last retrieved record in the previous step. 

For voucher export, value of DOCUMENT_DEF.SQL_LOAD_RECORDS column should be: 

Select rec.* from RECORDS recwhere rec.SOURCE_ID = <BILLING_SOURCE_ID> order by rec.id;

And value of DOCUMENT_DEF.SQL_RELOAD_RECORDS column should be: 

Select rec.* from RECORDS recwhere rec.SOURCE_ID = <BILLING_SOURCE_ID> and rec.id > <MAX_LOADED_ID> order by rec.id;

If there are 20000 records, then the JSONDocumentFiller uses the SQL queries configured in the DOCUMENT_DEF record as explained below: 

• The query from SQL_LOAD_RECORDS column is used to load the first batch of records (10000).
• If there are more records to be added to the same JSON file and the reload_method on one of the JSON_TAG_DEF entries is set to 'reloadRecords', the SQL from SQL_RELOAD_RECORDS column is used to load the next batches, with 10000 max rows selected in each subsequent batch, until all pending records have been added to the file.

3.5 - ValueFormatter

The value formatter is used in order to be able to configure the formatting of field values. Here a formatter class and a formatter_expression can be specified that will be used to generate a string.

Following Value Formatter types are available:

3.5.1 - Columns

3.5.2 - How to use VALUE_FORMATTER and JSON_DATA_TYPE to get desired result

It is possible to represent the field values in a JSON Document in different ways. If the value should be represented as a string, it will be surrounded by double quotes. If it has to be written to the JSON file as a Number, no double quotes will be placed around the value. If the value should be represented as a boolean, the boolean equivalent of the actual field value (i.e. true/false) will be written to the JSON file. Consider the following scenarios:

(here the term 'actual field value' refers to  the value obtained by calling a method or the value of the field from the Map object)

1. The actual field value is a Number type , but would like to write it as a String in the JSON file. For example : Actual field value = 100, but we want the value written to the JSON file to be : "100"
2. The actual field value is a Number with no decimals , but would like to write it as a float value in the JSON file. For example : Actual field value = 100, but we want the value written to the JSON file to be : 100.00
3. The actual field value is a String/Number, but would like to write it as a Boolean in the JSON file. For example : Actual field value = "1" / 1, but we want the value written to the JSON file to be : true
4. The actual field value is a String type and contains some characters that should be escaped to be JSON compatible. For example: Actual field value = "Value with "quotes"", but we want the value to be escaped properly before writing it to the JSON file.

During JSON file generation, the actual field value is always modified in this order, to get the desired result:

• First, the VALUE_FORMATTER corresponding to the JSON_TAG_DEF.FORMATTER_ID is applied to format the field value
• Then, JSON_TAG_DEF.JSON_DATA_TYPE configuration setting is used to convert the formatted field value to desired JSON type 

So let us take a look at the scenarios we mentioned above and see how they can be achieved with proper configuration:

4 - Example of Generation

After having configured a new JSON document definition and the tags structure it is possible to generate a JSON file with the following steps:

1. retrieve the configured DocumentDef

• docDef = (DocumentDef)getBroker().getInstance(DocumentDef.class,"key = " + <doc_key>);

2. create a context hash map containing the information needed by the filler to populate the dataBeans map.

3. create a new output stream used to write the JSON-file

4. call docDef.produce(context, outputStream);

Then the docDefwill generate the dataBeans map calling the filler associated with the DOCUMENT_DEF entry and will invoke the write method on the root tag of the JSON.

Then the generation will be propagated to all the key/value pairs will be written to the output stream according to the fields and methods set in the JSON_TAG_DEF table.