How to handle Large FlatFiles?

By using FFIterator we can handle large files

Large File Handling Techniques in webMethods

Lot's have doubts regarding large file handling using webMethods Integration tool, below is the process for handling:

In middleware integration we generally face a issue how to handle large data of flatfiles and xml . in the case of large files(suppose greater than 2MB) , it will slowdown the Integration server if directly loaded into the server , if the file is of huge size (greater than 100MB) then in this case it eill might crash the server . So, in order to save overself from this situation we must take a precautive measure, this can be like streaming the file into the server . So, in this blog we will see basically how to do flat file handling in Integaration server in the case of both Flat File and xml files.

Large File Handling (Flat Files)

Now we will see how to do large file handling in the case of flat files.

we can use webMethods FilePolling Port , pub.file:getFile , or using FTP (i.e using this service pub.client:ftp) to get the file depending on whether the file is on localsystem or remote FTP system w.r.t to webMethods Integration server(IS).

webMethods FilePolling port by default streams the file into the IS , for pub.file:getFile select the optionsal input parameter as load as stream and it will stream the file into the server .

Normally we use pub.flatFile:convertToValues service to convert the flatfile into IS document , this time also we will use the same service with slight modification we will set the iterate input parameter as true.

pub.flatFile:convertToValues: This service is used in convert the flat file data to IS Document. Iterate = true value ensures that the records from the flat file is read one By one. Output ffIterator is passed at the input of the next invocation of this service.

If ffIterator is null, exit from the loop. It means all the available records in the flat file are processed.

Large File Handling XML

Generally in real time scenarion for large xml it has been seen that some nodes are there that are being repeated several tomes making the xml file large enough . The large file handling for xml in webMethods targets this point and in built services are present to hanle these kind of scenarios.

Let's first see the inbuilt services present to do large file handling in the case of xml.

pub.xml:xmlStringToXMLNode – This service converts the input from String to XML node , this is a general service used in webMethods where we want to convert xml string data to xml node . This service is good in built service having great performance it plays a vital role in large file handling.

pub.xml:getXMLNodeIterator : This service will get the node one by one according to the Criteria set in the pipeline. Over here the criteria is set as EmployeeDetails and OFCDetails. This is the main service used for large file handling for XML , Just like flat file service, this service also iterates over the node and process it one by one .

pub.xml:getNextXMLNode : This service is used in order to get the next node.

pub.xml:xmlNodeToDocument : This service converts the XMLNode to Document.

Now lets see the structure of the main flow service where Large file Handling for xml has been done.

Like this we have handle large files... :)