XML Generation

Change log:

Description

The purpose of this document is to provide a description of the configuration of XML Cdr files, so that they can be read correctly by the Service Providers.

1 - XML Generation

It was required to export the CDR files and the records related to them in XML-format, and for this reason some generic utilities were created in Rator to generate XML.

This page describes the design of the framework and also some examples to use it in other parts of the system.

2 - Overview

The generation of an XML-file is done according to a database configuration where it is possible to configure all the tags that must be printed, their structure and how data can be retrieved to fill their content.

Once the configuration is ready the XML-producer just needs the data that will be passed as a map.

<File>
  <Name>M_OI_BIL.M_SP_CORR.GGSN101_20101125235414_14068.dat.SP_IN.038476.dat</Name>
  <BillingSourceId>201012031002438274</BillingSourceId>
  <Record>
    <BillingRecordId>201011262043130885</BillingRecordId>
    <RatingKey></RatingKey>
    <PriceElements>
      <PriceElement>
        <ChargeItemId>201005121347550887</ChargeItemId>
        <Description>SMS til 31240912</Description>
        <IdlId>201012011045262305</IdlId>
      </PriceElement>
    </PriceElements>
    <Code>HI3GNET</Code>
    <Duration>0</Duration>
  </Record>
</File>

In this example the map can contain 2 objects: “HEADER” will contain the billing source object used to fill the <Name> and the <BillingSourceId> tags, then in the map we will also find an object containing the list of records with key “RECORDS” for instance. The keys and the methods to retrieve the data from the map and from the objects inside the map will be specified in the configuration.

2.1 - Encoding

The encoding used to generate the file is UTF-8, therefore the XML-declaration of the generated XML-files will be "<?xml version="1.0" encoding="UTF-8"?>".

3 - Design

The persistent objects introduced in this framework and which are part of the configuration are XmlTagDef (table xml_tag_def) and XmlDocumentDef (xml_document_def) and ValueFormatter (value_formatter).

The valueFormatter can also be used outside the XML-generation.

3.1 - XmlTagDef

A record in the xml_tag_def defines a single tag of the resulting XML, and it has the following attributes:

3.1.1 - Columns

3.2 - XmlDocumentDef

An XmlDocumentDef models the definition for one type of XML. Here a filler class can be defined that will contain the logic to fill the map with the data, the methods to be called and also the reload ones.

3.2.1 - Columns

Note: Please note that the columns sql_load_records, sql_reload_records are added to the XmlDocumentDef class overridden within the FileProcessor Integration project as they are added to support the GenericXMLFiller implementation for XML-file export by FileProcessor engine. So these are available only from the FileProcessor integration project.

3.3 - XmlFiller

An abstract class XmlFiller has been defined with an abstract method populate that will generate the dataBeans map according to a context map.

This class contains the specific logic for the generation of the XML-document.

Concrete subclasses of xmlFiller can be found in the FileProcessor Integration project: see GenericXMLFiller, XmlFillerBillingRecords and XmlFillerIdl.

3.3.1 - GenericXMLFiller

The GenericXMLfiller class contains the logic for generating the XML-files from data obtained by joining some database tables by executing the SQL-script configured via the corresponding XML_DOCUMENT_DEF record.  If the required XML-file needs to execute multiple SQL-queries to get the data that is to be placed in the XML-document in different levels, the GenericXMLfiller cannot be used. A new subclass of  XmlFiller must be created, if the GenericXMLFiller cannot produce the XML-file in the required format.  

Instructions to set up file processor configuration for XML-file generation with GenericXMLFiller:

• Create XML_TAG_DEF entries for the required XML-file format: A record in the xml_tag_def defines a single tag of the resulting xml. It is possible to define parent tag for the XML_TAG_DEF entry, and thus it is possible to define the full hierarchy of the XML-elements.
  
  
• Create an XML_DOCUMENT_DEF record with FILLER_CLASS = 'com.CDRator.billing.dataloader.cdrdata.xml.GenericXMLFiller', XML_ROOT_TAG_ID = '<Id of the XML-Root Tag entry from XML_TAG_DEF entries created for this XML-file format>'
  
  
• The SQL_LOAD_RECORDS column of the above created XML_DOCUMENT_DEF record must contain the SQL-query to obtain the records that should be exported to the file. This SQL-string can contain the label <BILLING_SOURCE_ID> which will be replaced with appropriate value at run-time.
  If this SQL-query string contains the label <BILLING_SOURCE_ID>, it will be replaced with the ID of the current source record currently being processed to create the XML-file. Note: The billing source table name is configured in the column BILLING_SOURCE_TABLE_NAME of the table CDR_FILE_DEF. <BILLING_SOURCE_ID> will be replaced with the ID of the current row being processed from the table configured in the column CDR_FILE_DEF.BILLING_SOURCE_TABLE_NAME.
  
  
• The SQL_RELOAD_RECORDS column of the above created XML_DOCUMENT_DEF record must contain the SQL-query to load the next batch of records with ID greater than the last loaded maximum ID, which are to be exported into the XML-file.  This is used only if the reload_method is setup on at least one of the xml_tag_def entries. This SQL-query can contain the tags <BILLING_SOURCE_ID> and <MAX_LOADED_ID> labels which will be replaced with appropriate values at run-time.
  
  
• Make sure that the SQL-scripts setup in SQL_LOAD_RECORDS, SQL_RELOAD_RECORDS columns of the corresponding XML_DOCUMENT_DEF are valid.
  
  
• Create a CDR_FILE_DEF record with CODE ='XML' , BILLING_SOURCE_TABLE_NAME = <Source table name>, BILLING_SOURCE_CLASS= <Java class for Billing Source>, EXPORT_TYPE = '<The Export_Type for the Customer this XML-file is generated for>', XML_DOCUMENT_DEF_ID = '<ID of the XML_DOCUMENT_DEF record created in above step>'.
  
  
• Create a CDR_RECORD_DEF record with CODE ='DETAIL' , RECORD_TABLE_NAME = <DB table name for the records to be exported>, RECORD_CLASS = <Java class name for the Records to be exported>, CDR_FILE_DEF_ID = '<ID of the CDR_FILE_DEF created in above step>'. The XML-file processing engine will only need a CDR_RECORD_DEF entry for DETAIL records. So HEADER, TRAILER record definitions are not needed.
  
  
• Create a CDR_FILE_PROCESSOR entry with CODE='GENERATOR',  CDR_FILE_DEF_ID = '<ID of the CDR_FILE_DEF created in an earlier step>', FILE_TRANSMITTER_ID= '<Id of the corresponding FILE_TRANSMITTER entry>' and BRAND_ID = '<Corresponding brandId>'

Important Note: It is mandatory to configure both XML_DOCUMENT_DEF.SQL_LOAD_RECORDS and  XML_DOCUMENT_DEF.SQL_RELOAD_RECORDS columns with appropriate SQL-queries, because the File processor XML-filler classes are configured to load only a maximum of 10,000 records in one select. If there are more records in that database table, and we want to add them to the XML-file, the reload_method  column of the relevant xml_tag_def entry must be set to 'reloadRecords'. In that case, after printing the first set of 10,000 records, the GenericXMLFiller tries to load next batch of records with an ID greater than the ID of the last retrieved record in the previous step. 

Example value for XML_DOCUMENT_DEF.SQL_LOAD_RECORDS column can be:

Select rec.* from RECORDS recwhere rec.SOURCE_ID = <BILLING_SOURCE_ID> order by rec.id;

Example value for XML_DOCUMENT_DEF.SQL_RELOAD_RECORDS column can be:

Select rec.* from RECORDS recwhere rec.SOURCE_ID = <BILLING_SOURCE_ID> and rec.id > <MAX_LOADED_ID> order by rec.id;

If there are 20,000 records, then the GenericXMLFiller uses the SQL-queries configured in the corresponding XML_DOCUMENT_DEF entry like this:

• The query from SQL_LOAD_RECORDS column is used to load the first batch of records (10000) .
• If there are more records to be added to the same XML-file and the reload_method on one of the XML_TAG_DEF entries is set to 'reloadRecords', the SQL from SQL_RELOAD_RECORDS column is used to load the next batches, with 10000 max rows selected in each subsequent batch, until all pending records have been added to the file.

3.3.2 - XmlFillerBillingRecords and XmlFillerIdl

Detailed description of XmlFillerBillingRecords and XmlFillerIdl can be found at /wiki/spaces/Technology/pages/126782302  (internal link).

3.4 - ValueFormatter

The value formatter has been introduced in order to be able to configure the formatting of objects. Here a formatter class and a pattern can be specified that will be used to generate a string.

Two of them have been committed in Core, one for dates and one for numbers:

• com.CDRator.billing.utils.formatter.DateFormatter
• com.CDRator.billing.utils.formatter.NumberFormatter

3.4.1 - Columns

4 - Example of Generation

After having configured a new XML-definition and the tags structure it is possible to generate an XML with the following steps:

1. retrieve the configured xmlDocumentDef

• xmlDocDef = (XmlDocumentDef)getBroker().getInstance(XmlDocumentDef.class,"key = " + <doc_key>);

2. create a context hash map containing the information needed by the filler to populate the dataBeans map.

3. create a new output stream used to write the XML-file generated

4. call xmlDocDef.produce(context, outputStream);

Then the xmlDocDef will generate the dataBeans map calling the filler associated with the file and will invoke the write method on the root tag of the xml.

Then the generation will be propagated to all the sub tags, and the data will be filled according to the fields and methods set in the xml_tag_def table.