...
Date: | Author: | Version: | Changes: | Ext. | Int. | Is in Core | Jira Ref. | |
|---|---|---|---|---|---|---|---|---|
XX Month 20XX | Kilroy21 December 2010 | Davide Silvestre | 0.1 | Doc. created | x |
| N/A |
|
| 02 November 2015 | Karuna Thulluri02 November 2015 | Karuna Thulluri | 0.2 | Added details about GenericXMLFiller | N/A | |||
| 26 November 2015 | Karuna Thulluri | 0.3 | Updated Value Formatter details |
Description
The purpose of this document is to provide a description of the configuration of Xml XML Cdr files, so that they can be read correctly by the Service Providers.
1 -
...
XML Generation
For HI3G we needed It was required to export the CDR files and the records related to them in XML-format. For , and for this reason we have created some generic utilities were created in Rator to generate XML.
In this document we will describe This page describes the design of this the framework and also some examples to use it in other parts of the system.
2 - Overview
The generation of an xml XML-file is done according to a database configuration where it is possible to configure all the tags that must be printed, their structure and how we can retrieve the data data can be retrieved to fill their content.
Once the configuration is ready the Xml Producer XML-producer just needs the data , that will be passed as a Mapmap.
| Code Block | ||||
|---|---|---|---|---|
| ||||
<File>
<Name>M_OI_BIL.M_SP_CORR.GGSN101_20101125235414_14068.dat.SP_IN.038476.dat</Name>
<BillingSourceId>201012031002438274</BillingSourceId>
<Record>
<BillingRecordId>201011262043130885</BillingRecordId>
<RatingKey></RatingKey>
<PriceElements>
<PriceElement>
<ChargeItemId>201005121347550887</ChargeItemId>
<Description>SMS til 31240912</Description>
<IdlId>201012011045262305</IdlId>
</PriceElement>
</PriceElements>
<Code>HI3GNET</Code>
<Duration>0</Duration>
</Record>
</File>
|
In this example the map can contain 2 objects: “HEADER” will contain the billing source object used to fill the <Name> and the <BillingSourceId> tags, then in the map we will also find an object containing the list of records with key “RECORDS” for instance. The keys and the methods to retrieve the data from the map and from the objects inside the map will be specified in the configuration.
2.1 - Encoding
The encoding used to generate the file is UTF-8, therefore the xml XML-declaration of the generated xml XML-files will be "<?xml version="1.0" encoding="UTF-8"?>".
3 - Design
The persistent objects introduced in this framework and that which are part of the configuration are XmlTagDef (table xml_tag_def) and XmlDocumentDef (xml_document_def) and ValueFormatter (value_formatter).
The valueFormatter can also be used outside the xml XML-generation.
3.1 - XmlTagDef
A record in the xml_tag_def defines a single tag of the resulting xml XML, and it has the following attributes:
3.1.1 - Columns
...
| TAG_NAME |
...
| Name of the tag, in the previous example will be Record, Code, Name |
...
| MAP_KEY |
...
| Key to retrieve the correct component from the map. In the previous example this can be HEADER or RECORDS. |
...
| VALUE_TYPE |
...
| Can be FIELD (if the value to print is a field of the map object) or METHOD (if the content is retrieved calling a specific method in the map object). |
...
| FIELD_NAME |
...
| If the value type is field then this specify the field to print in the tag content. |
...
| METHOD_NAME |
...
| if the value type is method, then this specify the method to invoke in the map object to retrieve the data to fill the content for the current tag. |
...
| PARENT_ID |
...
| Defines the ID of the parent tag and therefore it is used to define the structure of the |
...
| XML-file. The parent |
...
| ID for the root tag will be null. |
...
| FLUSH_CONTENT |
...
| if ‘Y’ the content generated in the outputStream will be flushed in the |
...
| XML-file when the end tag of the current tag is printed. |
...
| FORMATTER_ID | ID |
...
| of the value_formatter table. This can be used to format the output data |
...
| (see |
...
| subsection below). |
...
| METHOD_INPUT |
...
| In some cases when the METHOD_NAME is invoked we also need some extra information that must be passed in the method. For instance in the cdr generation if we have a list of invoice detail lines inside the record tag we need to retrieve only the invoice detail lines related to that billing record and this can be done by defining in this field the tag name related to the object to pass: in this case it will be “Record”. |
...
| METHOD_INPUT_CLASS_NAME |
...
| This is the class name of the method_input field (“com.CDRator.billing.rating.BillingRecords” in this example). |
...
| SEQ_ORDER |
...
| If a tag has more than one child, here we can specify the preferred order to print them. |
...
| RELOAD_METHOD |
...
| If the tag to print is related to a list (i.e. a list of records) then it is possible to load them in different steps. In the |
...
| XML- generation when the content of the list has been printed then we try to retrieve further elements using the reload method if set. |
3.2 - XmlDocumentDef
An XmlDocumentDef models the definition for one type of xmlXML. Here we can define a filler class can be defined that will contain the logic to fill the map with the data, the methods to be called and also the reload ones.
3.2.1 - Columns
...
| DESCRIPTION | Short description of the document definition |
...
| KEY | Unique key to retrieve the document definition |
...
| FILLER_CLASS | Class name used to generate the map for this specific type of xml. This class will also be part of the map if it is needed to call methods inside the filler to retrieve data. |
| XML_ROOT_TAG_ID | Reference of the first tag of the XML. |
| SQL_LOAD_RECORDS |
...
| Contains the SQL-query to obtain the records that should be exported to the file. This SQL-string can contain the label <BILLING_SOURCE_ID> which will be replaced with appropriate value at run-time. | |
| SQL_RELOAD_RECORDS | Contains the SQL-query to load the next batch of records with ID greater than the last loaded maximum ID, which are to be exported into the XML-file. This is used only if the reload_method is setup on at least one of the xml_tag_def entries. This SQL-query can contain the tags <BILLING_SOURCE_ID> and <MAX_LOADED_ID> labels which will be replaced with appropriate values at run-time |
Note: Please note that the columns sql_load_records, sql_reload_records are added to the XmlDocumentDef class overridden within the FileProcessor Integration project as they are added to support the GenericXMLFiller implementation for XML-file export by FileProcessor engine. So these are available only from the FileProcessor integration project.
3.3 - XmlFiller
An abstract class XmlFiller has been defined with an abstract method populate that will generate the dataBeans map according to a context map.
In this This class is contained contains the specific logic for the generation of the xml XML-document.
Concrete subclasses of xmlFiller can be found in the FileProcessor Integration project: see GenericXMLFiller, XmlFillerBillingRecords and XmlFillerIdl.
3.3.1 - GenericXMLFiller
The GenericXMLfiller class contains the logic for generating the XML-files from data obtained by joining some database tables by executing the SQL-script configured via the corresponding XML_DOCUMENT_DEF record. If the required XML-file needs to execute multiple SQL-queries to get the data that is to be placed in the XML-document in different levels, the GenericXMLfiller cannot be used. A new subclass of XmlFiller need to must be created, if the GenericXMLFiller cannot produce the XML-file in the required format.
Instructions to set up file processor configuration for XML-file generation with GenericXMLFiller:
- Create XML_TAG_DEF entries for the required XML-file format: A record in the xml_tag_def defines a single tag of the resulting xml. It is possible to define parent tag for the XML_TAG_DEF entry, and thus it is possible to define the full hierarchy of the XML-elements.
- Create an XML_DOCUMENT_DEF record with FILLER_CLASS = 'com.CDRator.billing.dataloader.cdrdata.xml.GenericXMLFiller', XML_ROOT_TAG_ID = '<Id of the XML-Root Tag entry from XML_TAG_DEF entries created for this XML-file format>'
- The SQL_LOAD_RECORDS column of the above created XML_DOCUMENT_DEF record must contain the SQL-query to obtain the records that should be exported to the file. This SQL-string can contain the label <BILLING_SOURCE_ID> which will be replaced with appropriate value at run-time.
If this SQL-query string contains the label <BILLING_SOURCE_ID>, it will be replaced with the ID of the current source record currently being processed to create the XML-file. Note: The billing source table name is configured in the column BILLING_SOURCE_TABLE_NAME of the table CDR_FILE_DEF. <BILLING_SOURCE_ID> will be replaced with the ID of the current row being processed from the table configured in the column CDR_FILE_DEF.BILLING_SOURCE_TABLE_NAME. - The SQL_RELOAD_RECORDS column of the above created XML_DOCUMENT_DEF record must contain the SQL-query to load the next batch of records with ID greater than the last loaded maximum ID, which are to be exported into the XML-file. This is used only if the reload_method is setup on at least one of the xml_tag_def entries. This SQL-query can contain the tags <BILLING_SOURCE_ID> and <MAX_LOADED_ID> labels which will be replaced with appropriate values at run-time.
- Make sure that the SQL-scripts setup in SQL_LOAD_RECORDS, SQL_RELOAD_RECORDS columns of the corresponding XML_DOCUMENT_DEF are valid.
- Create a CDR_FILE_DEF record with CODE ='XML' , BILLING_SOURCE_TABLE_NAME = <Source table name>, BILLING_SOURCE_CLASS= <Java class for Billing Source>, EXPORT_TYPE = '<The Export_Type for the Customer this XML-file is generated for>', XML_DOCUMENT_DEF_ID = '<ID of the XML_DOCUMENT_DEF record created in above step>'.
- Create a CDR_RECORD_DEF record with CODE ='DETAIL' , RECORD_TABLE_NAME = <DB table name for the records to be exported>, RECORD_CLASS = <Java class name for the Records to be exported>, CDR_FILE_DEF_ID = '<ID of the CDR_FILE_DEF created in above step>'. The XML-file processing engine will only need a CDR_RECORD_DEF entry for DETAIL records. So HEADER, TRAILER record definitions are not needed.
- Create a CDR_FILE_PROCESSOR entry with CODE='GENERATOR', CDR_FILE_DEF_ID = '<ID of the CDR_FILE_DEF created in an earlier step>', FILE_TRANSMITTER_ID= '<Id of the corresponding FILE_TRANSMITTER entry>' and BRAND_ID = '<Corresponding brandId>'
- Make sure that the parameter tree entries for XML generation are created and contain proper values: See Parameter Tree entries for XML generation with GenericXMLFiller
Important Note: It is mandatory to configure both XML_DOCUMENT_DEF.SQL_LOAD_RECORDS and XML_DOCUMENT_DEF.SQL_RELOAD_RECORDS columns with appropriate SQL-queries, because the File processor XML-filler classes are configured to load only a maximum 10000 of 10,000 records in one select. If there are more records in that database table, and we want to add them to the XML-file, the reload_method column of the relevant xml_tag_def entry must be set to 'reloadRecords'. In that case, after printing the first set of 10000 10,000 records, the GenericXMLFiller tries to load next batch of records , having the with an ID greater than the ID of the last retrieved record in the previous step.
Example value for XML_DOCUMENT_DEF.SQL_LOAD_RECORDS column can be:
| Code Block | ||
|---|---|---|
| ||
Select |
...
rec.* |
...
from RECORDS recwhere rec.SOURCE_ID = <BILLING_SOURCE_ID> order by |
...
rec.id; |
Example value for XML_DOCUMENT_DEF.SQL_RELOAD_RECORDS column can be:
| Code Block | ||
|---|---|---|
| ||
Select |
...
rec.* |
...
from RECORDS recwhere rec.SOURCE_ID = <BILLING_SOURCE_ID> and |
...
rec.id > <MAX_LOADED_ID> order by |
...
rec.id; |
If there are 20000 20,000 records, then the GenericXMLFiller uses the SQL-queries configured in the above parameter tree entries corresponding XML_DOCUMENT_DEF entry like this:
- The query from RECORDSQL_RETRIEVELOAD_SQL parameter RECORDS column is used to load the first batch of records (10000) .
- If there are more records to be added to the same XML-file and the reload_method on one of the XML_TAG_DEF entries is set to 'reloadRecords', The the SQL from RECORD_RETRIEVE_SQL_WITH_ID_GREATERTHAN_CLAUSE parameter RELOAD_RECORDS column is used to load the next batches, with 10000 max rows selected in each subsequent batch, until all pending records have been added to the file.
3.3.2 - XmlFillerBillingRecords and XmlFillerIdl
Detailed description of XmlFillerBillingRecords and XmlFillerIdl can be found here: Xml Cdr Exporterat /wiki/spaces/Technology/pages/126782302 (internal link).
3.4 - ValueFormatter
The value formatter has been introduced in order to be able to configure the formatting of objects. Here we can specify a formatter class and a pattern can be specified that will be used to generate a string.
In the core I have committed two of them: Two of them have been committed in Core, one for dates and one for numbers:-
- com.CDRator.billing.utils.formatter.DateFormatter
...
- com.CDRator.billing.utils.formatter.NumberFormatter
3.4.1 - Columns
| NAME |
- name: formatter name
- code: formatter code
...
| Formatter name | |
| CODE | Formatter code |
| DESCRIPTION | Describes the purpose of the formatter |
...
| VALUE_CLASS | Defines the input for the formatter (i.e. java.util.Date or java.lang.Number |
...
or java.lang.String) Important Note: The formatter should be chosen based on the actual value type. If the value type is not compatible with the java type defined in VALUE_FORMATTER.VALUE_CLASS, the formatting fails and hence file generation fails.
| |
| FORMATTER_CLASS | Class used to format (i.e. com.CDRator.billing.utils.formatter.DateFormatter) |
...
| FORMATTER_METHOD | Usually it is format |
...
| FORMATTER_EXPRESSION | Pattern used to convert the object |
...
to formatted string |
...
. This is optional for the formatters that do not require a pattern to be defined. Examples:
|
...
|
...
| |
| NULL_VALUE | Value returned if the input object is null. This is optional and defaults to null. |
4 - Example of Generation
After having configured a new xml XML-definition and the tags structure it is possible to generate an xml XML with the following steps:
1. retrieve the configured xmlDocumentDef
- xmlDocDef = (XmlDocumentDef)getBroker().getInstance(XmlDocumentDef.class,"key = " + <doc_key>);
2. create a context hash map containing the information needed by the filler to populate the dataBeans map.
3. create a new output stream used to write the xml XML-file generated
4. call xmlDocDef.produce(context, outputStream);
Then the xmlDocDef will generate the dataBeans map calling the filler associated to with the file and will invoke the write method on the root tag of the xml.
Then the generation will be propagated to all the sub tags, and the data will be filled according to the fields and methods set in the xml_tag_def table.