poi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wessel van Norel <delgu...@gmail.com>
Subject OutOfMemoryError while writing a large xlsx file
Date Mon, 24 Jan 2011 19:05:42 GMT
Hello all,

Today I tried to change my HSSFWorkbook based code to XSSFWorkbook,
because the sheet I'm generating needs to contain more then 65k rows
(about 99k now). The last HSSFWorkbook I created was about 20M in
size. Knowing xmlx is XML based I expected writing the XSSFWorkbook
would take longer and need more memory, but I didn't expect it to need
> 2Gb of memory to be generated (my heap space is currently > 4Gb but
quite some is used by other parts of the application that is
generating the xlsx file). It needs even more memory and didn't
complete the writing process so I can't give you any detailed
information about the end result (unfortunately the code that
generates the XML depends on lots of EJB objects, so I can't do it
without the JBoss application server).

I read about the BigGridDemo:
http://www.docjar.org/html/api/org/apache/poi/xssf/usermodel/examples/BigGridDemo.java.html

And I found the following post:
http://apache-poi.1045710.n5.nabble.com/Yet-another-quot-Out-of-memory-quot-td3316218.html

So with the last post in mind I hope I can give Nick (and all others
on this list) a bit more detail, so hopefully this problem can be
resolved before I have to use the BigGridDemo, which would take a
considerable rewrite of my Excel exporting code.

I've created a 4.5Gb heapdump and let the eclipse memory analyzer do
it's trick. I've attached a few screenshots of the results, hoping
that these contain enough information for you. I'm not at liberty to
share the heapdump.

The biggest object is: org.apache.poi.xssf.usermodel.XSSFSheet. This
sheet is 2Gb in size. It contains almost 2 million
org.apache.xmlbeans.impl.store.Xobj$ElementXobj objects, totalling
1.7Gb in heap space retained.

The XSSFCells total 180M of memory, still a lot, but not anywhere near the 2Gb.


Unfortunately I've no experience, yet, with writing XML files using
XMLBeans. So I'm not sure if it's doing it the SAX or DOM way when
writing XML files. But looking at the memory usage, my first guess
would be that it's doing it the DOM way.


If you have anymore questions which can be answered by looking in
different ways at the heapdump, don't hesitate to ask.

I hope one of you is able to help me out here, or at least point me
where I need to poke in the POI code to change the way this XML file
is being written.

Thanks in advance.

Regards,
Wessel van Norel


Mime
View raw message