Groovy render chunk file - file

Question

I am using StreamingMarkupBuilder for generate very big xml file. File is so big, that i have set
JAVA_OPT="-XX:NewSize=2048m -XX:MaxNewSize=4096m -XX:SurvivorRatio=8 -Xms4096m -Xmx4096m -XX:MaxPermSize=4096m"
(2048m, 1024m not enough)
Without chunk i got file so far, response answer:
*0:0:0:0:0:0:0:1 - - [03/Jan/2018:17:15:37 +0200] "GET ... HTTP/1.1" 200 167467288*
Service:
def builder = new StreamingMarkupBuilder()
def Header = {
mkp.xmlDeclaration()
mkp.declareNamespace('': 'http://...')
...
...
Header{
SourceDocuments {
SalesInvoices {
// !!!big scope of data came in this loop!!!
for (row in data) {
Invoice {
...
...
}
}
}
}
}
XmlUtil.serialize(builder.bind(Header))
NOTE: Only builder.bind(Header) as last result line create file very slowly.
Controller:
response.contentType = grailsApplication.config.grails.mime.types[exportFormat]
response.setHeader("Content-disposition", "attachment; filename=" + reportName + "." + exportFormat)
response.outputStream << result
I am looking for chunk solution for response file and minimize JAVA_OPT environments heaps sizes. Actually to get a solution with DOM would be perfect because of performance, and I would like appreciate that, but will be nice if solution of chunk exists on different way that have groovy.

Answer 1

this answer is not about chunking but about performance
here are 4 examples of rendering xml with time result comparison
note that minimal memory consumption will be with StreamingMarkupBuilder because it renders xml directly to output (file) without building it in memory
in other cases we are building xml in memory and then rendering to output
the fastest one will be with CompileStatic annotation for the main loop
START: StreamingMarkupBuilder
END: 23946 sec
START: DOMBuilder
END: 8282 sec
START: NodeBuilder
END: 8778 sec
START: NodeBuilder+CompileStatic
END: 2978 sec
code:
import groovy.util.Node
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
class Const{
static long MAX_ITEMS=999999
}
//=============StreamingMarkupBuilder==========================
long t=System.currentTimeMillis()
println "START: StreamingMarkupBuilder"
def xdata = new groovy.xml.StreamingMarkupBuilder().bind{
root {
for(int i=0;i<Const.MAX_ITEMS;i++){
a('id':i,'123_'+i)
}
}
}
new File('c:/11/111.xml').withOutputStream{stream->
stream.withWriter("UTF-8"){ w-> xdata.writeTo( w ) }
}
println " END: ${System.currentTimeMillis()-t} sec"
xdata=null
System.gc()
Thread.sleep(5000)
//=============DOMBuilder======================================
t=System.currentTimeMillis()
println "START: DOMBuilder"
DocumentBuilderFactory domFactory=DocumentBuilderFactory.newInstance();
domFactory.setNamespaceAware(true);
DocumentBuilder domBuilder = domFactory.newDocumentBuilder();
xdata = new groovy.xml.DOMBuilder(domBuilder).
root{
for(int i=0;i<Const.MAX_ITEMS;i++){
a('id':i,'123_'+i)
}
}
new File('c:/11/112.xml').withOutputStream{stream->
groovy.xml.XmlUtil.serialize(xdata, stream)
}
println " END: ${System.currentTimeMillis()-t} sec"
xdata=null
System.gc()
Thread.sleep(5000)
//=============NodeBuilder=====================================
t=System.currentTimeMillis()
println "START: NodeBuilder"
xdata = new NodeBuilder().
root{
for(int i=0;i<Const.MAX_ITEMS;i++){
a('id':i,'123_'+i)
}
}
new File('c:/11/113.xml').withOutputStream{stream->
groovy.xml.XmlUtil.serialize(xdata, stream)
}
println " END: ${System.currentTimeMillis()-t} sec"
xdata=null
System.gc()
Thread.sleep(5000)
//=============NodeBuilder+CompileStatic=======================
t=System.currentTimeMillis()
println "START: NodeBuilder+CompileStatic"
#groovy.transform.CompileStatic
def longLoopData(Node parent){
for(int i=0;i<Const.MAX_ITEMS;i++){
parent.appendNode('a', ['id':i], '123_'+i)
}
}
xdata = new NodeBuilder().
root{
longLoopData(current)
}
new File('c:/11/114.xml').withOutputStream{stream->
groovy.xml.XmlUtil.serialize(xdata, stream)
}
println " END: ${System.currentTimeMillis()-t} sec"
xdata=null
System.gc()
Thread.sleep(5000)