XML Generation

Change log:

Date:	Author:	Version:	Changes:	Ext.	Int.	Is in Core	Jira Ref.
21 December 2010	Davide Silvestre	0.1	Doc. created	x		N/A
02 November 2015	Karuna Thulluri	0.2	Added details about GenericXMLFiller			N/A

Description

The purpose of this document is to provide a description of the configuration of XML Cdr files, so that they can be read correctly by the Service Providers.

1 - XML Generation

It was required to export the CDR files and the records related to them in XML-format, and for this reason some generic utilities were created in Rator to generate XML.

This page describes the design of the framework and also some examples to use it in other parts of the system.

2 - Overview

The generation of an XML-file is done according to a database configuration where it is possible to configure all the tags that must be printed, their structure and how data can be retrieved to fill their content.

Once the configuration is ready the XML-producer just needs the data that will be passed as a map.

<File>
  <Name>M_OI_BIL.M_SP_CORR.GGSN101_20101125235414_14068.dat.SP_IN.038476.dat</Name>
  <BillingSourceId>201012031002438274</BillingSourceId>
  <Record>
    <BillingRecordId>201011262043130885</BillingRecordId>
    <RatingKey></RatingKey>
    <PriceElements>
      <PriceElement>
        <ChargeItemId>201005121347550887</ChargeItemId>
        <Description>SMS til 31240912</Description>
        <IdlId>201012011045262305</IdlId>
      </PriceElement>
    </PriceElements>
    <Code>HI3GNET</Code>
    <Duration>0</Duration>
  </Record>
</File>

In this example the map can contain 2 objects: “HEADER” will contain the billing source object used to fill the <Name> and the <BillingSourceId> tags, then in the map we will also find an object containing the list of records with key “RECORDS” for instance. The keys and the methods to retrieve the data from the map and from the objects inside the map will be specified in the configuration.

2.1 - Encoding

The encoding used to generate the file is UTF-8, therefore the XML-declaration of the generated XML-files will be "<?xml version="1.0" encoding="UTF-8"?>".

3 - Design

The persistent objects introduced in this framework and which are part of the configuration are XmlTagDef (table xml_tag_def) and XmlDocumentDef (xml_document_def) and ValueFormatter (value_formatter).

The valueFormatter can also be used outside the XML-generation.

3.1 - XmlTagDef

A record in the xml_tag_def defines a single tag of the resulting XML, and it has the following attributes:

3.1.1 - Columns

TAG_NAME	Name of the tag, in the previous example will be Record, Code, Name
MAP_KEY	Key to retrieve the correct component from the map. In the previous example this can be HEADER or RECORDS.
VALUE_TYPE	Can be FIELD (if the value to print is a field of the map object) or METHOD (if the content is retrieved calling a specific method in the map object).
FIELD_NAME	If the value type is field then this specify the field to print in the tag content.
METHOD_NAME	if the value type is method, then this specify the method to invoke in the map object to retrieve the data to fill the content for the current tag.
PARENT_ID	Defines the ID of the parent tag and therefore it is used to define the structure of the XML-file. The parent ID for the root tag will be null.
FLUSH_CONTENT	if ‘Y’ the content generated in the outputStream will be flushed in the XML-file when the end tag of the current tag is printed.
FORMATTER_ID	ID of the value_formatter table. This can be used to format the output data (see subsection below).
METHOD_INPUT	In some cases when the METHOD_NAME is invoked we also need some extra information that must be passed in the method. For instance in the cdr generation if we have a list of invoice detail lines inside the record tag we need to retrieve only the invoice detail lines related to that billing record and this can be done by defining in this field the tag name related to the object to pass: in this case it will be “Record”.
METHOD_INPUT_CLASS_NAME	This is the class name of the method_input field (“com.CDRator.billing.rating.BillingRecords” in this example).
SEQ_ORDER	If a tag has more than one child, here we can specify the preferred order to print them.
RELOAD_METHOD	If the tag to print is related to a list (i.e. a list of records) then it is possible to load them in different steps. In the XML- generation when the content of the list has been printed then we try to retrieve further elements using the reload method if set.

3.2 - XmlDocumentDef

An XmlDocumentDef models the definition for one type of XML. Here a filler class can be defined that will contain the logic to fill the map with the data, the methods to be called and also the reload ones.

3.2.1 - Columns

DESCRIPTION	Short description of the document definition
KEY	Unique key to retrieve the document definition
FILLER_CLASS	Class name used to generate the map for this specific type of xml. This class will also be part of the map if it is needed to call methods inside the filler to retrieve data.
XML_ROOT_TAG_ID	Reference of the first tag of the XML.
SQL_LOAD_RECORDS	Contains the SQL-query to obtain the records that should be exported to the file. This SQL-string can contain the label <BILLING_SOURCE_ID> which will be replaced with appropriate value at run-time.
SQL_RELOAD_RECORDS	Contains the SQL-query to load the next batch of records with ID greater than the last loaded maximum ID, which are to be exported into the XML-file. This is used only if the reload_method is setup on at least one of the xml_tag_def entries. This SQL-query can contain the tags <BILLING_SOURCE_ID> and <MAX_LOADED_ID> labels which will be replaced with appropriate values at run-time

Note: Please note that the columns sql_load_records, sql_reload_records are added to the XmlDocumentDef class overridden within the FileProcessor Integration project as they are added to support the GenericXMLFiller implementation for XML-file export by FileProcessor engine. So these are available only from the FileProcessor integration project.

3.3 - XmlFiller

An abstract class XmlFiller has been defined with an abstract method populate that will generate the dataBeans map according to a context map.

This class contains the specific logic for the generation of the XML-document.

Concrete subclasses of xmlFiller can be found in the FileProcessor Integration project: see GenericXMLFiller, XmlFillerBillingRecords and XmlFillerIdl.

3.3.1 - GenericXMLFiller

The GenericXMLfiller class contains the logic for generating the XML-files from data obtained by joining some database tables by executing the SQL-script configured via the corresponding XML_DOCUMENT_DEF record. If the required XML-file needs to execute multiple SQL-queries to get the data that is to be placed in the XML-document in different levels, the GenericXMLfiller cannot be used. A new subclass of XmlFiller must be created, if the GenericXMLFiller cannot produce the XML-file in the required format.

Instructions to set up file processor configuration for XML-file generation with GenericXMLFiller:

Create XML_TAG_DEF entries for the required XML-file format: A record in the xml_tag_def defines a single tag of the resulting xml. It is possible to define parent tag for the XML_TAG_DEF entry, and thus it is possible to define the full hierarchy of the XML-elements.
Create an XML_DOCUMENT_DEF record with FILLER_CLASS = 'com.CDRator.billing.dataloader.cdrdata.xml.GenericXMLFiller', XML_ROOT_TAG_ID = '<Id of the XML-Root Tag entry from XML_TAG_DEF entries created for this XML-file format>'
The SQL_LOAD_RECORDS column of the above created XML_DOCUMENT_DEF record must contain the SQL-query to obtain the records that should be exported to the file. This SQL-string can contain the label <BILLING_SOURCE_ID> which will be replaced with appropriate value at run-time.
If this SQL-query string contains the label <BILLING_SOURCE_ID>, it will be replaced with the ID of the current source record currently being processed to create the XML-file. Note: The billing source table name is configured in the column BILLING_SOURCE_TABLE_NAME of the table CDR_FILE_DEF. <BILLING_SOURCE_ID> will be replaced with the ID of the current row being processed from the table configured in the column CDR_FILE_DEF.BILLING_SOURCE_TABLE_NAME.
The SQL_RELOAD_RECORDS column of the above created XML_DOCUMENT_DEF record must contain the SQL-query to load the next batch of records with ID greater than the last loaded maximum ID, which are to be exported into the XML-file. This is used only if the reload_method is setup on at least one of the xml_tag_def entries. This SQL-query can contain the tags <BILLING_SOURCE_ID> and <MAX_LOADED_ID> labels which will be replaced with appropriate values at run-time.
Make sure that the SQL-scripts setup in SQL_LOAD_RECORDS, SQL_RELOAD_RECORDS columns of the corresponding XML_DOCUMENT_DEF are valid.
Create a CDR_FILE_DEF record with CODE ='XML' , BILLING_SOURCE_TABLE_NAME = <Source table name>, BILLING_SOURCE_CLASS= <Java class for Billing Source>, EXPORT_TYPE = '<The Export_Type for the Customer this XML-file is generated for>', XML_DOCUMENT_DEF_ID = '<ID of the XML_DOCUMENT_DEF record created in above step>'.
Create a CDR_RECORD_DEF record with CODE ='DETAIL' , RECORD_TABLE_NAME = <DB table name for the records to be exported>, RECORD_CLASS = <Java class name for the Records to be exported>, CDR_FILE_DEF_ID = '<ID of the CDR_FILE_DEF created in above step>'. The XML-file processing engine will only need a CDR_RECORD_DEF entry for DETAIL records. So HEADER, TRAILER record definitions are not needed.
Create a CDR_FILE_PROCESSOR entry with CODE='GENERATOR', CDR_FILE_DEF_ID = '<ID of the CDR_FILE_DEF created in an earlier step>', FILE_TRANSMITTER_ID= '<Id of the corresponding FILE_TRANSMITTER entry>' and BRAND_ID = '<Corresponding brandId>'

Important Note: It is mandatory to configure both XML_DOCUMENT_DEF.SQL_LOAD_RECORDS and XML_DOCUMENT_DEF.SQL_RELOAD_RECORDS columns with appropriate SQL-queries, because the File processor XML-filler classes are configured to load only a maximum of 10,000 records in one select. If there are more records in that database table, and we want to add them to the XML-file, the reload_method column of the relevant xml_tag_def entry must be set to 'reloadRecords'. In that case, after printing the first set of 10,000 records, the GenericXMLFiller tries to load next batch of records with an ID greater than the ID of the last retrieved record in the previous step.

Example value for XML_DOCUMENT_DEF.SQL_LOAD_RECORDS column can be:

Select vr.* from VOUCHER_EXPORT_RECORD vr where vr.VOUCHER_EXPORT_SOURCE_ID = <BILLING_SOURCE_ID> order by vr.id

Example value for XML_DOCUMENT_DEF.SQL_RELOAD_RECORDS column can be:

Select vr.* from VOUCHER_EXPORT_RECORD vr where vr.VOUCHER_EXPORT_SOURCE_ID = <BILLING_SOURCE_ID> and vr.id > <MAX_LOADED_ID> order by vr.id

If there are 20,000 records, then the GenericXMLFiller uses the SQL-queries configured in the above parameter tree entries like this:

The query from RECORD_RETRIEVE_SQL parameter is used to load the first batch of records (10000) .
If there are more records to be added to the same XML-file and the reload_method on one of the XML_TAG_DEF entries is set to 'reloadRecords', the SQL from RECORD_RETRIEVE_SQL_WITH_ID_GREATERTHAN_CLAUSE parameter is used to load the next batches, with 10000 max rows selected in each subsequent batch, until all pending records have been added to the file.

3.3.2 - XmlFillerBillingRecords and XmlFillerIdl

Detailed description of XmlFillerBillingRecords and XmlFillerIdl can be found at /wiki/spaces/Technology/pages/126782302 (internal link).

3.4 - ValueFormatter

The value formatter has been introduced in order to be able to configure the formatting of objects. Here a formatter class and a pattern can be specified that will be used to generate a string.

Two of them have been committed in Core, one for dates and one for numbers:

com.CDRator.billing.utils.formatter.DateFormatter
com.CDRator.billing.utils.formatter.NumberFormatter

3.4.1 - Columns

NAME	Formatter name
CODE	Formatter code
DESCRIPTION	Describes the purpose of the formatter
VALUE_CLASS	Defines the input for the formatter (i.e. java.util.Date or java.lang.Number)
FORMATTER_CLASS	Class used to format (i.e. com.CDRator.billing.utils.formatter.DateFormatter)
FORMATTER_METHOD	Usually it is format
FORMATTER_EXPRESSION	Pattern used to convert the object in the string (i.e. “dd.MM.yyyy” or “#,##0.00”)
NULL_VALUE	Value returned if the input object is null.

4 - Example of Generation

After having configured a new XML-definition and the tags structure it is possible to generate an XML with the following steps:

1. retrieve the configured xmlDocumentDef

xmlDocDef = (XmlDocumentDef)getBroker().getInstance(XmlDocumentDef.class,"key = " + <doc_key>);

2. create a context hash map containing the information needed by the filler to populate the dataBeans map.

3. create a new output stream used to write the XML-file generated

4. call xmlDocDef.produce(context, outputStream);

Then the xmlDocDef will generate the dataBeans map calling the filler associated with the file and will invoke the write method on the root tag of the xml.

Then the generation will be propagated to all the sub tags, and the data will be filled according to the fields and methods set in the xml_tag_def table.