Change log:

Date:	Author:	Version:	Changes:	Ext.	Int.	Is in Core	Jira Ref.
25 November 2015	Karuna Thulluri	0.1	JSON File export feature added to File Processor (Not yet completed)			N/A

Description

The purpose of this document is to provide a description of the JSON file export feature of the File Processor framework.

1 - JSON Generation

To be able to export data from Rator database tables into a JSON file, some generic utilities were created in Rator File Processor framework.

This page describes the design of the framework and also some examples to use it in other parts of the system.

2 - Overview

The generation of a JSON-file is done according to a database configuration where it is possible to configure all the tags that must be printed, their structure and how data can be retrieved to fill their content.

Once the configuration is ready the JSON-producer just needs the data that will be passed as a map.

Sample JSON file

{"voucher_file" : {"currency" : "XYZ","amount" : "200",
"vouchers" : {
"voucher" : {"id" : "11111","serial_number" : "11111","secret_code" : "11111"}
"voucher" : {"id" : "22222","serial_number" : "22222","secret_code" : "22222"}
"voucher" : {"id" : "33333","serial_number" : "33333","secret_code" : "33333"}
"voucher" : {"id" : "44444","serial_number" : "44444","secret_code" : "44444"}
"voucher" : {"id" : "55555","serial_number" : "55555","secret_code" : "55555"}
}
}
}

In this example the map can contain 2 objects: “HEADER” will contain the billing source object used to fill the "currency" and "amount" key/value pairs. The keys and the methods to retrieve the data from the map and from the objects inside the map will be specified in the configuration table called JSON_TAG_DEF.

2.1 - Encoding

The encoding used to generate the file is UTF-8..

3 - Design

The persistent objects introduced in this framework as part of the configuration are JSONTagDef (table JSON_TAG_DEF) and DocumentDef (table DOCUMENT_DEF). The configuration also uses the ValueFormatter (table VALUE_FORMATTER).

The valueFormatter can also be used outside the JSON-generation.

3.1 - Data model

3.1.1 - JSON File model

The following tables are presented in the data model diagram below:

CDR_FILE_DEF (class CdrFileDef)
CDR_RECORD_DEF (class CdrRecordDef)
DOCUMENT_DEF(class DocumentDef)
JSON_TAG_DEF (class JSONTagDef)
VALUE_FORMATTER (class ValueFormatter)

All the above mentioned classes are persistent.

3.1.2 - File Processor Model

3.2 - JSONTagDef

A record in the JSON_TAG_DEF defines a single tag (key/value pair) of the resulting JSON file, and it has the following attributes:

3.2.1 - Columns

TAG_NAME

Name of the tag. For example currency, amount, Name, Voucher, Id, brand etc

MAP_KEY

Key to retrieve the correct component from the map. Example: HEADER or FILEDEF or RECORD_DEF or tag name of the parent record from which a child tag value can be obtained.

HEADER holds the file source record.

FILEDEF holds the current instance of JSONDocumenrFiller.

RECORD_DEF holds the CDR_RECORD_DEF (with CODE = DETAIL) instance set up for this JSON Document export

VALUE_TYPE

Can be FIELD (if the value to print is a field of the map object) or METHOD (if the content is retrieved calling a specific method in the map object obtained via the key set in MAP_KEY).

FIELD_NAME

If the value type is FIELD then this specify the field to print in the JSON tag content.

METHOD_NAME

if the value type is METHOD, then this specify the method to invoke on the map object to retrieve the data to fill the content for the current tag.

JSON_DATA_TYPE

The data type with which the JSON field value should be represented while writing value to file. Possible value are OBJECT, ARRAY, STRING, NUMBER andBOOLEAN.

OBJECT, ARRAY types can be used only for the parent tag types (i.e JSON_TAG_DEF entries that have children).
STRING, NUMBER, BOOLEAN types can be used only for the JSON_TAG_DEF entries that do not have any children.

Please see: Value Formatter and How to use VALUE_FORMATTER and JSON_DATA_TYPE for more details

Note: JSON Key names will always be surrounded by double quotes. This JSON_DATA_TYPE configuration settings are applicable only for values, not the keys.

JSON_DATA_TYPE

value

Description of behavior

Example

OBJECT (Default setting for the JSON_TAG_DEF entries that have children)

Can be used only for the parent tag types.

If this type is chosen, the children of this parent tag will be surrounded by curly braces { }

JSON Tag Voucher is set up as OBJECT type

JSON_TAG_DEF.TAG_NAME = voucher, JSON_DATA_TYPE = OBJECT / empty
Resulting line in JSON file:
"voucher" : { "id" : "12345","serial_number" : "12345","secret_code" : "12345" }

ARRAY

Can be used only for the parent tag types.

If this type is chosen, the children of this parent tag will be surrounded by square brackets [ ]

JSON Tag Voucher is set up as OBJECT type

JSON_TAG_DEF.TAG_NAME = voucher, JSON_DATA_TYPE = ARRAY
Resulting line in JSON file:
"voucher" : [ "id" : "12345","serial_number" : "12345","secret_code" : "12345" ]

STRING (Default setting for JSON_TAG_DEF entries that do not have any children)

Can be used only for the JSON_TAG_DEF entries that do not have any children.

Resulting value will be surrounded by double quotes.

JSON_TAG_DEF.TAG_NAME = brand

Actual Field Value

Result After Choosing

JSON_DATA_TYPE = STRING

Resulting key value pair in JSON file

XYZ

"XYZ"

"brand" : "XYZ"

NUMBER

Can be used only for the JSON_TAG_DEF entries that do not have any children.

Resulting value will not have double quotes around. Should be used for presenting numeric type values in JSON file.

JSON_TAG_DEF.TAG_NAME = count

Actual Field Value

Result After Choosing

JSON_DATA_TYPE = NUMBER

Resulting key value pair in JSON file

43

43

"count" : 43

BOOLEAN

Can be used only for the JSON_TAG_DEF entries that do not have any children.

Boolean equivalent of the actual field value gets written to the json file. Resulting value will not be surrounded by double quotes

JSON_TAG_DEF.TAG_NAME = exported

Actual Field Value	Result After Choosing JSON_DATA_TYPE = BOOLEAN	Resulting key value pair in JSON file
1	true	"exported" : true
0	false	"exported" : false
abcdefg	false	"exported" : false
Y	true	"exported" : true
True	true	"exported" : true

PARENT_ID

Defines the ID of the parent tag and therefore it is used to define the structure of the JSON-file. The parent ID for the root tag will be null.

FLUSH_CONTENT

if ‘Y’ the content generated in the outputStream will be flushed in the JSON-file when the end tag of the current tag is printed.

FORMATTER_ID

ID of appropriate VALUE_FORMATTER table entry that is suitable to format this field value. The value formatter used to format the JSON field value, before applying the JSON_DATA_TYPE. So, first the Value formatter is applied to the field value, and then the JSON_DATA_TYPE setting is applied.

Please see: Value Formatter and How to use VALUE_FORMATTER and JSON_DATA_TYPE for more details

METHOD_INPUT

In some cases when the METHOD_NAME is invoked we also need some extra information that must be passed in the method. For instance in the cdr generation if we have a list of invoice detail lines inside the record tag we need to retrieve only the invoice detail lines related to that billing record and this can be done by defining in this field the tag name related to the object to pass: in this case it will be “Record”.

METHOD_INPUT_CLASS_NAME

This is the class name of the method_input field (“com.CDRator.billing.rating.BillingRecords” for example).

SEQ_ORDER

If a tag has more than one child, here we can specify the preferred order to print them.

RELOAD_METHOD

If the tag to print is related to a list (i.e. a list of records) then it is possible to load them in multiple batches. In the JSON file generation when the content of the current batch has been printed then we try to retrieve further elements using the reload method if set.

3.3 - DocumentDef

An DocumentDef models the definition for a document type for example JSON. Here a filler class can be defined that will contain the logic to fill the map with the data, the methods to be called and also the reload method.

3.3.1 - Columns

DESCRIPTION	Short description of the document definition
KEY	Unique key to retrieve the document definition
FILLER_CLASS	Class name used to generate the map for this specific type of document. This class will also be part of the map (with key "FILEDEF") if it is needed to call methods inside the filler to retrieve data.
ROOT_TAG_ID	Reference of the Root tag of the document.
SQL_LOAD_RECORDS	Contains the SQL-query to obtain the records that should be exported to the file. This SQL-query string can contain the label <BILLING_SOURCE_ID> which will be replaced with appropriate value at run-time.
SQL_RELOAD_RECORDS	Contains the SQL-query to load the next batch of records with ID greater than the last loaded maximum ID, which are to be exported into the JSON-file. This is used only if the reload_method is setup on at least one of the JSON_TAG_DEF entries. This SQL-query can contain the tags <BILLING_SOURCE_ID> and <MAX_LOADED_ID> labels which will be replaced with appropriate values at run-time

3.4 - DocumentFiller

An abstract class DocumentFiller has been defined with an abstract method populate that will generate the dataBeans map according to a context map.

The concrete sub-classes of DocumentFiller class should contain the specific logic for the generation of the document.

Following Concrete sub-classes of DocumentFiller can be found in the FileProcessor Integration project. See JSONDocumentFiller for more details

3.4.1 - JSONDocumentFiller

The JSONDocumentFiller class contains the logic for generating the JSON-files from data obtained by joining some database tables by executing the SQL-script configured via the corresponding DOCUMENT_DEF record. If the desired JSON-file format cannot be achieved with JSONDocumentFiller sub class, A new subclass of DocumentFiller must be created as per the requirements.

Instructions to set up file processor configuration for JSON-file generation with JSONDocumentFiller :

Create JSON_TAG_DEF entries for the required JSON file format: A record in the JSON_TAG_DEF defines a single element (key/value pair) of the resulting JSON file. It is possible to define parent tag for the JSON_TAG_DEF entry, and thus it is possible to define the full hierarchy of the JSON elements. Please choose proper FORMATTER_ID (ID of the VALUE_FORMATTER table record, used to format the value before writing it to the JSON file) for each JSON_TAG_DEF entry based on the value type.
Create a DOCUMENT_DEF record with FILLER_CLASS = 'com.CDRator.billing.document.json.JSONDocumentFiller', ROOT_TAG_ID = '<Id of the JSON Root Tag entry from JSON_TAG_DEF entries created for this JSON file format>'
The SQL_LOAD_RECORDS column of the above created DOCUMENT_DEF record must contain the SQL query to obtain the records that should be exported to the file. This SQL string can contain the label <BILLING_SOURCE_ID> which will be replaced with appropriate value at run-time.
If this SQL query string contains the label <BILLING_SOURCE_ID>, it will be replaced with the ID of the current source record currently being processed to create the JSON file. Note: The billing source table name is configured in the column BILLING_SOURCE_TABLE_NAME of the table CDR_FILE_DEF. <BILLING_SOURCE_ID> will be replaced with the ID of the current row being processed from the table configured in the column CDR_FILE_DEF.BILLING_SOURCE_TABLE_NAME.
The SQL_RELOAD_RECORDS column of the above created DOCUMENT_DEF record must contain the SQL query to load the next batch of records with ID greater than the last loaded maximum ID, which are to be exported into the JSON file. This SQL query can contain the tags <BILLING_SOURCE_ID> and <MAX_LOADED_ID> labels which will be replaced with appropriate values at run-time
Create a CDR_FILE_DEF record with CODE ='JSON' , BILLING_SOURCE_TABLE_NAME =<Source table name>, BILLING_SOURCE_CLASS= <Java class for Billing Source>, EXPORT_TYPE = '<The Export_Type for the Customer this JSON file is generated for>', DOCUMENT_DEF_ID = '<ID of the DOCUMENT_DEF record created in above step>', FILE_NAME_SUFFIX = 'json'. Please note that the CDR_PARSER_ID, RECORD_SEPARATOR_ID are not used for JSON file export and hence they can be left empty.
A post processor method name, post processor hookpoint key, filename format and/or filename generator method can be configured on this CDR_FILE_DEF as needed.
Create a CDR_RECORD_DEF record with CODE ='DETAIL' , RECORD_TABLE_NAME = <DB table name for the records to be exported>, RECORD_CLASS = <Java class name for the Records to be exported>, CDR_FILE_DEF_ID = '<ID of the CDR_FILE_DEF created in above step>'. The JSON file processing engine will only need a CDR_RECORD_DEF entry for DETAIL records. So HEADER, TRAILER record definitions are not needed.
Create a CDR_FILE_PROCESSOR entry with CODE='GENERATOR', CDR_FILE_DEF_ID = '<ID of the CDR_FILE_DEF created in earlier step>', FILE_TRANSMITTER_ID= '<Id of the corresponding FILE_TRANSMITTER entry>' and BRAND_ID = '<Corresponding brandId>'

Important Note: It is mandatory to configure both DOCUMENT_DEF.SQL_LOAD_RECORDS and DOCUMENT_DEF.SQL_RELOAD_RECORDS columns with appropriate SQL queries, because the File processor JSON filler class is configured to load only a maximum 10000 records in one select. If there are more records in that Voucher Batch, the reload_method column of the relevant JSON_TAG_DEF entry must be set to 'reloadRecords'. In that case, after printing the first set of 10000 records, the JSONDocumentFiller tries to load next batch of records with IDs greater than the ID of the last retrieved record in the previous step.

For voucher export, value of DOCUMENT_DEF.SQL_LOAD_RECORDS column should be:

Select rec.* from RECORDS recwhere rec.SOURCE_ID = <BILLING_SOURCE_ID> order by rec.id;

And value of DOCUMENT_DEF.SQL_RELOAD_RECORDS column should be:

Select rec.* from RECORDS recwhere rec.SOURCE_ID = <BILLING_SOURCE_ID> and rec.id > <MAX_LOADED_ID> order by rec.id;

If there are 20000 records, then the JSONDocumentFiller uses the SQL queries configured in the DOCUMENT_DEF record as explained below:

The query from SQL_LOAD_RECORDS column is used to load the first batch of records (10000).
If there are more records to be added to the same JSON file and the reload_method on one of the JSON_TAG_DEF entries is set to 'reloadRecords', the SQL from SQL_RELOAD_RECORDS column is used to load the next batches, with 10000 max rows selected in each subsequent batch, until all pending records have been added to the file.

3.5 - ValueFormatter

The value formatter is used in order to be able to configure the formatting of field values. Here a formatter class and a formatter_expression can be specified that will be used to generate a string.

Following Value Formatter types are available:

Value Formatter implementation	Description	Example
com.CDRator.billing.utils.formatter.DateFormatter	Used to format a Date object to a String according the formatter_expression set on this Value_Formatter entry.	Input value Type: Date formatter_expression: dd.MM.yyyy Input value: 26-11-2015 10:00:00 Result: 26.11.2015
com.CDRator.billing.utils.formatter.NumberFormatter	Used to format a Number to a String according the formatter_expression set on this Value_Formatter entry.	Input value Type: any number type formatter_expression: #,##0.00 Input value: 200 Result: 200.00
com.CDRator.billing.utils.formatter.XmlStringFormatter	Used to escape a String to be compatible with XML format	Input value Type: String Input value: x>y Result: x>y
com.CDRator.billing.utils.formatter.JsonStringFormatter	Used to escape a String to be compatible with JSON format	Input value Type: String Input value: 'test' Result: \'test\'

3.5.1 - Columns

NAME	A unique name to distinguish different Formatters
CODE	Formatter code
DESCRIPTION	Describes the purpose of the formatter
VALUE_CLASS	Defines the input for the formatter (i.e. java.util.Date or java.lang.Number or java.lang.String) Important Note: The formatter should be chosen based on the actual value type. If the value type is not compatible with the java type defined in VALUE_FORMATTER.VALUE_CLASS, the formatting fails and hence file generation fails. If the value to be formatted is a String, then use a Value_Formatter that has the VALUE_CLASS = 'java.lang.String' If the value to be formatted is a Number type (int or long or Amount or float or Integer etc..), then use a Value_Formatter that has the VALUE_CLASS = 'java.lang.Number' If the value to be formatter is a Date, then use a Value_Formatter that has the VALUE_CLASS = 'java.util.Date'
FORMATTER_CLASS	Class used to format ( For example: com.CDRator.billing.utils.formatter.DateFormatter)
FORMATTER_METHOD	Method name from the class defined in FORMATTER_CLASS, that should be executed to format the input value. Usually it is 'format'.
FORMATTER_EXPRESSION	Pattern used to convert the object to formatted string. This is optional for the formatters that do not require a pattern to be defined. Examples: “dd.MM.yyyy” to format a Date object to a string in this format “#,##0.00” to format a Number to decimal format String
NULL_VALUE	Value returned if the input object is null. This is optional and defaults to null.

3.5.2 - How to use VALUE_FORMATTER and JSON_DATA_TYPE to get desired result

It is possible to represent the field values in a JSON Document in different ways. If the value should be represented as a string, it will be surrounded by double quotes. If it has to be written to the JSON file as a Number, no double quotes will be placed around the value. If the value should be represented as a boolean, the boolean equivalent of the actual field value (i.e. true/false) will be written to the JSON file. Consider the following scenarios:

(here the term 'actual field value' refers to the value obtained by calling a method or the value of the field from the Map object)

The actual field value is a Number type , but would like to write it as a String in the JSON file. For example : Actual field value = 100, but we want the value written to the JSON file to be : "100"
The actual field value is a Number with no decimals , but would like to write it as a float value in the JSON file. For example : Actual field value = 100, but we want the value written to the JSON file to be : 100.00
The actual field value is a String/Number, but would like to write it as a Boolean in the JSON file. For example : Actual field value = "1" / 1, but we want the value written to the JSON file to be : true
The actual field value is a String type and contains some characters that should be escaped to be JSON compatible. For example: Actual field value = "Value with "quotes"", but we want the value to be escaped properly before writing it to the JSON file.

During JSON file generation, the actual field value is always modified in this order, to get the desired result:

First, the VALUE_FORMATTER corresponding to the JSON_TAG_DEF.FORMATTER_ID is applied to format the field value
Then, JSON_TAG_DEF.JSON_DATA_TYPE configuration setting is used to convert the formatted field value to desired JSON type

So let us take a look at the scenarios we mentioned above and see how they can be achieved with proper configuration:

Scenario

Configuration

The actual field value is a Number type, but would like to write it as a String in the JSON file.

For example : Actual field value = 100, but we want the value written to the JSON file to be : "100"

Value Formatter: No formatter (or) Number formatter with formatter_expression containing no decimals

JSON_DATA_TYPE: STRING

The actual field value is a Number with no decimals, but would like to write it as a float value in the JSON file.

For example : Actual field value = 100, but we want the value written to the JSON file to be : 100.00

Value Formatter: Number formatter with formatter_expression to convert to decimal format

JSON_DATA_TYPE: NUMBER

The actual field value is a String, but would like to write it as a Boolean in the JSON file.

For example : Actual field value = "1"/1, but we want the value written to the JSON file to be : true

Value Formatter: No formatter needed

JSON_DATA_TYPE: BOOLEAN

The actual field value is a String type and contains some characters that should be escaped to be JSON compatible.

For example: Actual field value = "Value with "quotes"", but we want the value to be escaped properly before writing it to the JSON file.

Value Formatter: JsonStringFormatter

JSON_DATA_TYPE: STRING

Actual field value = "Value with "quotes""

Result: "Value with \"quotes\""

4 - Example of Generation

After having configured a new JSON document definition and the tags structure it is possible to generate a JSON file with the following steps:

1. retrieve the configured DocumentDef

docDef = (DocumentDef)getBroker().getInstance(DocumentDef.class,"key = " + <doc_key>);

2. create a context hash map containing the information needed by the filler to populate the dataBeans map.

3. create a new output stream used to write the JSON-file

4. call docDef.produce(context, outputStream);

Then the docDefwill generate the dataBeans map calling the filler associated with the DOCUMENT_DEF entry and will invoke the write method on the root tag of the JSON.

Then the generation will be propagated to all the key/value pairs will be written to the output stream according to the fields and methods set in the JSON_TAG_DEF table.

Browser not supported