poi-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Serra <mse...@iwu.edu>
Subject Re: AW: AW: AW: large xls files
Date Mon, 21 Feb 2005 20:55:32 GMT
Thank you for the fix, but it still doesn't load.  I now see two 
different threads, which together take up 98% CPU time.  Is there 
something else I'm supposed to change (besides what you wrote in the 
code)?  Does it work on your machine?

andy@superlinksoftware.com wrote:

> Fix inline
>
> Mike Serra wrote:
>
>> Sorry to make this so long-winded, but I've narrowed the code down to 
>> a specific test, and it's so simple I'm curious to know what could be 
>> going on.  Can anybody tell me why this code won't work:
>>
>> import java.io.*;
>> import org.apache.poi.hssf.usermodel.HSSFWorkbook;
>>
>> public class LoadTest {
>>
>>  public static void main(String[] args) {
>>
>>    HSSFWorkbook workbook;
>>    File inputFile = new File("big_test.xls");
>>    try {
>>    InputStream fileIn = new FileInputStream(inputFile);
>
>
>       fileIn = new BufferedInputStream(fileIn);
>
>>
>>    workbook = new HSSFWorkbook(fileIn);
>>    fileIn.close();
>>    } catch (FileNotFoundException ex) {
>>    } catch (IOException ex) { }
>>  }
>> }
>>
>> big_test.xls is a 500kb file, with no formatted text or anything 
>> unusual.  When I run the above code, I can follow /proc/meminfo and 
>> watch my ram slowly and continually decrease.  Also, the JVM takes up 
>> 98% CPU time.  This goes on forever.  What could possibly cause it?
>>
>> KHZ (SAW) wrote:
>>
>>> Hi Mike.
>>>
>>> I hope you find it soon. In a few days I'm away some weeks for 
>>> relaxing.
>>>
>>> Regards,    Karl-Heinz.
>>>
>>>
>>> -----Urspr?ngliche Nachricht-----
>>> Von: Mike E. Serra [mailto:mserra@iwu.edu] Gesendet: Montag, 21. 
>>> Februar 2005 18:04
>>> An: POI Users List; karl-heinz.zengerle@sawag.com
>>> Betreff: Re: AW: AW: large xls files
>>>
>>>
>>> I think what I'll try, is to get the POI source and step the 
>>> debugger into the function, and see what happens.  I'll let you know 
>>> (or post to the devel list) if anything interesting comes up.
>>> Thanks,
>>> Mike S.
>>>
>>> KHZ (SAW) wrote:
>>>
>>>
>>>  
>>>
>>>> Hi Mike.
>>>>
>>>> It sounds like a POI bug.
>>>>
>>>> But this could also be a result of software not fitting together. As
>>>> it's not running on two platforms the latter case or overload problems
>>>> in one system seem less likely.
>>>>
>>>> You could look for similar questions in the archive or the bug 
>>>> list. Ad
>>>> hoc I don't remember such an error.
>>>>
>>>> Regards,    Karl-Heinz.
>>>>
>>>>
>>>> -----Urspr?ngliche Nachricht-----
>>>> Von: Mike Serra [mailto:mserra@iwu.edu] Gesendet: Montag, 21. 
>>>> Februar 2005 17:47
>>>> An: POI Users List
>>>> Betreff: Re: AW: large xls files
>>>>
>>>> I have found the problem line, and it is this:
>>>>
>>>> workbook = new HSSFWorkbook(fileIn);
>>>>
>>>> The debugger steps into this line, and doesn't come out of it.
>>>> "Top" reports 98% CPU usage (I'm running linux 2.6), but this also 
>>>> happens on WinXP.
>>>> So, it sounds like an infinite loop, but, why would something loop 
>>>> infinitely for large files,
>>>> but work fine for small ones?
>>>> I should also note that this is not on or off behavior - for a 
>>>> medium sized file, the load takes a medium amount of time, say, 
>>>> thirty seconds.  But enlarge the file, and load time quickly grows 
>>>> to infinite.  It's as if the load time increases exponentially.
>>>>
>>>> KHZ (SAW) wrote:
>>>>
>>>>  
>>>>
>>>>> Hi Mike.
>>>>>
>>>>> We had files with around 3 MB which could be processed.
>>>>>
>>>>> The memory exception should come quite quickly. Possibly it's an
>>>>> infinite loop (in Windows: processor usage near 100 %, mainly used by
>>>>>     
>>>>
>>>>
>>> 1
>>>  
>>>
>>>>> process). You've to be sure to put the -Xmx option to the right tool.
>>>>> E.g. for debugging in Eclipse it's at at different place than for
>>>>> running jsp pages using Tomcat.
>>>>>
>>>>> Regards,    Karl-Heinz.
>>>>>
>>>>>
>>>>> -----Urspr?ngliche Nachricht-----
>>>>> Von: Mike Serra [mailto:mserra@iwu.edu] Gesendet: Montag, 21. 
>>>>> Februar 2005 17:24
>>>>> An: POI Users List
>>>>> Betreff: Re: large xls files
>>>>>
>>>>> My thanks to you and Karl-Heinz for your JVM tuning tips, but 
>>>>> unfortunately I still can't open the file.  I have tried upping both
>>>>>     
>>>>
>>>>
>>>> the
>>>>  
>>>>
>>>>> max heap size and the initial heap.  Maybe I'm just not being 
>>>>> patient enough - I think the file might load eventually but I've 
>>>>> given it over
>>>>>     
>>>>
>>>>
>>>
>>>  
>>>
>>>>> ten minutes with no luck so far.  Tell me just one thing: even 
>>>>> when memory usage is high, is it normal for POI to spend a long time
>>>>>     
>>>>
>>>>
>>> loading
>>>  
>>>
>>>>> a file?  If it isn't, the problem must be with my code.
>>>>>
>>>>> andy@superlinksoftware.com wrote:
>>>>>
>>>>>
>>>>>
>>>>>    
>>>>>
>>>>>> I have no problems with 1000+ row files.   The problem is likely

>>>>>> that
>>>>>>       
>>>>>
>>>>>
>>>
>>>  
>>>
>>>>>> you're likely running out of heap.  java -Xmx128mb (for example)
>>>>>>
>>>>>> There are plans for doing this as NIO mem-mapped but its merely a

>>>>>> matter of funding.  This plus the conversion to primatives 
>>>>>> representing NUMBERs would make probably an 8-fold memory 
>>>>>> decrease (I'd guess).
>>>>>>
>>>>>> -andy
>>>>>>
>>>>>> Mike Serra wrote:
>>>>>>
>>>>>>  
>>>>>>      
>>>>>>
>>>>>>> Hello out there to all POI users,
>>>>>>> I am writing a Java app which runs programmable macros on xls

>>>>>>> spreadsheets (project AutoSpreadsheet on sourceforge), and I

>>>>>>> find that large Excel files, with 1000 rows or more, simply do

>>>>>>> not load.
>>>>>>>         
>>>>>>
>>>>>>
>>>
>>>  
>>>
>>>>>>> I understand that there are memory issues with POI, but I didn't

>>>>>>> think it was that bad.  Is there any workaround for this?  If

>>>>>>> performance with POI-Ruby would be significantly better, I might

>>>>>>> simply port the whole app to Ruby.  Has anyone had any 
>>>>>>> experience with this?
>>>>>>>
>>>>>>> --------------------------------------------------------------------

>>>>>>>
>>>>>>>         
>>>>>>
>>>>>>
>>> -
>>>  
>>>
>>>>>>> To unsubscribe, e-mail: poi-user-unsubscribe@jakarta.apache.org
>>>>>>> Mailing List:     http://jakarta.apache.org/site/mail2.html#poi
>>>>>>> The Apache Jakarta Poi Project:  http://jakarta.apache.org/poi/
>>>>>>>
>>>>>>>            
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------

>>>>>>
>>>>>> To unsubscribe, e-mail: poi-user-unsubscribe@jakarta.apache.org
>>>>>> Mailing List:     http://jakarta.apache.org/site/mail2.html#poi
>>>>>> The Apache Jakarta Poi Project:  http://jakarta.apache.org/poi/
>>>>>>
>>>>>>
>>>>>>  
>>>>>>       
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: poi-user-unsubscribe@jakarta.apache.org
>>>>> Mailing List:     http://jakarta.apache.org/site/mail2.html#poi
>>>>> The Apache Jakarta Poi Project:  http://jakarta.apache.org/poi/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: poi-user-unsubscribe@jakarta.apache.org
>>>>> Mailing List:     http://jakarta.apache.org/site/mail2.html#poi
>>>>> The Apache Jakarta Poi Project:  http://jakarta.apache.org/poi/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>     
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: poi-user-unsubscribe@jakarta.apache.org
>>>> Mailing List:     http://jakarta.apache.org/site/mail2.html#poi
>>>> The Apache Jakarta Poi Project:  http://jakarta.apache.org/poi/
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: poi-user-unsubscribe@jakarta.apache.org
>>>> Mailing List:     http://jakarta.apache.org/site/mail2.html#poi
>>>> The Apache Jakarta Poi Project:  http://jakarta.apache.org/poi/
>>>>
>>>>
>>>>   
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: poi-user-unsubscribe@jakarta.apache.org
>>> Mailing List:     http://jakarta.apache.org/site/mail2.html#poi
>>> The Apache Jakarta Poi Project:  http://jakarta.apache.org/poi/
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: poi-user-unsubscribe@jakarta.apache.org
>>> Mailing List:     http://jakarta.apache.org/site/mail2.html#poi
>>> The Apache Jakarta Poi Project:  http://jakarta.apache.org/poi/
>>>
>>>
>>>  
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: poi-user-unsubscribe@jakarta.apache.org
>> Mailing List:     http://jakarta.apache.org/site/mail2.html#poi
>> The Apache Jakarta Poi Project:  http://jakarta.apache.org/poi/
>> .
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: poi-user-unsubscribe@jakarta.apache.org
> Mailing List:     http://jakarta.apache.org/site/mail2.html#poi
> The Apache Jakarta Poi Project:  http://jakarta.apache.org/poi/
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: poi-user-unsubscribe@jakarta.apache.org
Mailing List:     http://jakarta.apache.org/site/mail2.html#poi
The Apache Jakarta Poi Project:  http://jakarta.apache.org/poi/


Mime
View raw message