Xml Generation

Change log:

Description

The purpose of this document is to provide a description of the configuration of Xml Cdr files, so that they can be read correctly by the Service Providers.

1- Xml Generation

For HI3G we needed to export the CDR files and the records related to them in XML format. For this reason we have created some generic utilities in Rator to generate XML.

In this document we will describe the design of this framework and also some examples to use it in other parts of the system.

2 - Overview

The generation of an xml file is done according to a database configuration where it is possible to configure all the tags that must be printed, their structure and how we can retrieve the data to fill their content.

Once the configuration is ready the Xml Producer just needs the data, that will be passed as a Map.

<File>
  <Name>M_OI_BIL.M_SP_CORR.GGSN101_20101125235414_14068.dat.SP_IN.038476.dat</Name>
  <BillingSourceId>201012031002438274</BillingSourceId>
  <Record>
    <BillingRecordId>201011262043130885</BillingRecordId>
    <RatingKey></RatingKey>
    <PriceElements>
      <PriceElement>
        <ChargeItemId>201005121347550887</ChargeItemId>
        <Description>SMS til 31240912</Description>
        <IdlId>201012011045262305</IdlId>
      </PriceElement>
    </PriceElements>
    <Code>HI3GNET</Code>
    <Duration>0</Duration>
  </Record>
</File>

In this example the map can contain 2 objects: “HEADER” will contain the billing source object used to fill the <Name> and the <BillingSourceId> tags, then in the map we will also find an object containing the list of records with key “RECORDS” for instance. The keys and the methods to retrieve the data from the map and from the objects inside the map will be specified in the configuration.

2.1 - Encoding

The encoding used to generate the file is UTF-8, therefore the xml declaration of the generated xml files will be "<?xml version="1.0" encoding="UTF-8"?>".

3 - Design

The persistent objects introduced in this framework and that are part of the configuration are XmlTagDef (table xml_tag_def) and XmlDocumentDef (xml_document_def) and ValueFormatter (value_formatter).

The valueFormatter can also be used outside the xml generation.

3.1 - XmlTagDef

A record in the xml_tag_def defines a single tag of the resulting xml and it has the following attributes:

3.1.1 - Columns

- TAG_NAME: the name of the tag, in the previous example will be Record, Code, Name …                                                                                                                                                            -  MAP_KEY: this is the key to retrieve the correct component from the map. In the previous example this can be HEADER or RECORDS.

- VALUE_TYPE: this can be FIELD (if the value to print is a field of the map object) or METHOD (if the content is retrieved calling a specific method in the map object)

 - FIELD_NAME: if the value type is field then this specify the field to print in the tag content

 - METHOD_NAME: if the value type is method then this specify the method to invoke in the map object to retrieve the data to fill the content for the current tag.

- PARENT_ID: this defines the id of the parent tag and therefore it is used to define the structure of the xml file. The parent id for the root tag will be null.

- FLUSH_CONTENT: if ‘Y’ the content generated in the outputStream will be flushed in the xml file when the end tag of the current tag is printed.

- FORMATTER_ID: this is the id of the value_formatter table. This can be used to format the output data. (see section below)

 - METHOD_INPUT: In some cases when the METHOD_NAME is invoked we also need some extra information that must be passed in the method. For instance in the cdr generation if we have a list of invoice detail lines inside the record tag we need to retrieve only the invoice detail lines related to that billing record and this can be done by defining in this field the tag name related to the object to pass: in this case it will be “Record”.

- METHOD_INPUT_CLASS_NAME: This is the class name of the method_input field (“com.CDRator.billing.rating.BillingRecords” in this example)

- SEQ_ORDER: If a tag has more than one child, here we can specify the preferred order to print them.

- RELOAD_METHOD: If the tag to print is related to a list (i.e. a list of records) then it is possible to load them in different steps. In the xml generation when the content of the list has been printed then we try to retrieve further elements using the reload method if set.

3.2 - XmlDocumentDef

An XmlDocumentDef models the definition for one type of xml. Here we can define a filler class that will contain the logic to fill the map with the data, the methods to be called and also the reload ones.

3.2.1 - Columns

- description: short description of the document definition

- key: unique key to retrieve the document definition

- filler_class: class name used to generate the map for this specific type of xml. This class will also be part of the map if it is needed to call methods inside the filler to retrieve data.

- xml_root_tag_id: this is the reference of the first tag of the xml.

3.3 - XmlFiller

An abstract class XmlFiller has been defined with an abstract method populate that will generate the dataBeans map according to a context map.

In this class is contained the specific logic for the generation of the xml document.

Concrete subclasses of xmlFiller can be found in the FileProcessor Integration project: see GenericXMLFiller, XmlFillerBillingRecords and XmlFillerIdl.

3.3.1 - GenericXMLFiller

The GenericXMLfiller class contains the logic for creating an XML file from a list of records obtained by executing the SQL script configured via Parameter tree entries. This class can be used for generating the XML files from data obtained by joining some database tables by a query, configured in parameter tree as described below. If the required XML file needs to execute multiple SQL queries to get the data that is to be placed in the XML document in different levels, the GenericXMLfiller cannot be used. A new subclass of  XmlFiller need to be created, if the GenericXMLFiller cannot produce the XML file in required format.  

Parameter Tree entries used to configure the SQL script(s) used by GenericXMLFiller class:

Important Note: It is mandatory to setup both parameter tree entries RECORD_RETRIEVE_SQL and RECORD_RETRIEVE_SQL_WITH_ID_GREATERTHAN_CLAUSE, because the File processor XML filler classes are configured to load only a maximum 10000 records in one select. If there are more records in that database table, and we want to add them to the XML file, the reload_method  column of the relevant xml_tag_def entry must be set to 'reloadRecords'. In that case, after printing the first set of 10000 records, the GenericXMLFiller tries to load next batch of records, having the ID greater than the ID of the last retrieved record in the previous step. 

 For example: If there are 20000 records, then the GenericXMLFiller uses the SQL queries configured in the above parameter tree entries like this:

• The query from RECORD_RETRIEVE_SQL parameter is used to load the first batch of records (10000) .
• If there are more records to be added to the same XML file and the reload_method on one of the XML_TAG_DEF entries is set to 'reloadRecords', The SQL from RECORD_RETRIEVE_SQL_WITH_ID_GREATERTHAN_CLAUSE parameter is used to load the next batches , with 10000 max rows selected in each subsequent batch, until all pending records have been added to the file.

Instructions to set up file processor configuration for XML file generation with GenericXMLFiller:

• Create XML_TAG_DEF entries for the required XML file format: A record in the xml_tag_def defines a single tag of the resulting xml. It is possible to define parent tag for the XML_TAG_DEF entry, and thus it is possible to define the full hierarchy of the XML elements. 
• Create an XML_DOCUMENT_DEF record with FILLER_CLASS = 'com.CDRator.billing.dataloader.cdrdata.xml.GenericXMLFiller', XML_ROOT_TAG_ID = '<Id of the XML Root Tag entry from XML_TAG_DEF entries created for this XML file format>' 
• Create a CDR_FILE_DEF record with CODE ='XML' , BILLING_SOURCE_TABLE_NAME = <Source table name>, BILLING_SOURCE_CLASS= <Java class for Billing Source>, EXPORT_TYPE = '<The Export_Type for the Customer this XML file is generated for>', XML_DOCUMENT_DEF_ID = '<ID of the XML_DOCUMENT_DEF record created in above step>'.
• Create a CDR_RECORD_DEF record with CODE ='DETAIL' , RECORD_TABLE_NAME = <DB table name for the records to be exported>, RECORD_CLASS = <Java class name for the Records to be exported>, CDR_FILE_DEF_ID = '<ID of the CDR_FILE_DEF created in above step>'. The XML file processing engine will only need a CDR_RECORD_DEF entry for DETAIL records. So HEADER, TRAILER record definitions are not needed.
• Create a CDR_FILE_PROCESSOR entry with CODE='GENERATOR',  CDR_FILE_DEF_ID = '<ID of the CDR_FILE_DEF created in earlier step>', FILE_TRANSMITTER_ID= '<Id of the corresponding FILE_TRANSMITTER entry>' and BRAND_ID = '<Corresponding brandId>'
• Make sure that the parameter tree entries for XML generation are created and contain proper values: See 

3.3.2 - XmlFillerBillingRecords and XmlFillerIdl

Detailed description of XmlFillerBillingRecords and XmlFillerIdl can be found here: Xml Cdr Exporter

3.4 - ValueFormatter

The value formatter has been introduced in order to be able to configure the formatting of objects. Here we can specify a formatter class and a pattern that will be used to generate a string.

In the core I have committed two of them: one for dates and one for numbers:

- com.CDRator.billing.utils.formatter.DateFormatter

- com.CDRator.billing.utils.formatter.NumberFormatter

3.4.1 - Columns

- name: formatter name

- code: formatter code

- description: describes the purpose of the formatter

- value_class: defines the input for the formatter (i.e. java.util.Date or java.lang.Number)

- formatter_class: class used to format (i.e. com.CDRator.billing.utils.formatter.DateFormatter)

- formatter_method: usually it is format

- formatter_expression: pattern used to convert the object in the string (i.e. “dd.MM.yyyy” or “#,##0.00”)

- null_value: value returned if the input object is null.

4 - Example of Generation

After having configured a new xml definition and the tags structure it is possible to generate an xml with the following steps

1. retrieve the configured xmlDocumentDef

xmlDocDef = (XmlDocumentDef)getBroker().getInstance(XmlDocumentDef.class,"key = " + <doc_key>);

2. create a context hash map containing the information needed by the filler to populate the dataBeans map.

3. create a new output stream used to write the xml file generated

4. call xmlDocDef.produce(context, outputStream);

Then the xmlDocDef will generate the dataBeans map calling the filler associated to the file and will invoke the write method on the root tag of the xml.

Then the generation will be propagated to all the sub tags and the data will be filled according to the fields and methods set in the xml_tag_def table.