lucene-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From luc...@digiatlas.org
Subject Re: problems with large Lucene index
Date Thu, 12 Mar 2009 14:30:09 GMT
Thanks Mike and Jokin for your comments on the memory problem. I have  
submitted the query to the Hibernate Search list although I haven't  
seen a response yet.

In the meantime I did my own investigating in the code (I'd rather  
have avoided this!). I'm seeing results that don't make any sense and  
maybe someone here with more experience of Lucene and the way memory  
is allocated by the JVM may shed light on, what to me, are quite  
illogical observations.

As you may recall I had a stand-alone Lucene search and a Hibernate  
Search version.  Looking in the HS code did not shed any light on the  
issue. I took my stand-alone Lucene code and put it in a method and  
replaced the search in the HS class (the constructor of  
QueryHits.java) with the call to my method. Bear in mind this method  
is the same code as posted in my earlier message - it sets up the  
Lucene search from scratch (i.e.: no data structures created by HS  
were used). So, effectively I was calling my stand-alone code after  
any setup done by Hibernate and any memory it may have allocated  
(which turned out to be a few Mb).

I get OOM!  Printing the free memory at this point shows bags of  
memory left. Indeed, the same free memory (+/- a few Mb) as the  
stand-alone Lucene version!

I then instrumented the Lucene method where the OOM is occuring  
(FSDirectory.readInternal()).  I cannot understand the results I am  
seeing. Below is a snippet of the output of each with the code around  
FSDirectory line 598 as follows:

...
         do {
		long tot = Runtime.getRuntime().totalMemory();
		long free =Runtime.getRuntime().freeMemory();
System.out.println("LUCENE: offset="+offset+"  total="+total+"   
len-total="+(len-total)+"   free mem="+free+"  used ="+(tot-free));
           int i = file.read(b, offset+total, len-total);
...



The stand-alone version:

...
LUCENE: offset=0  total=0  len-total=401   free mem=918576864  used =330080544
LUCENE: offset=0  total=0  len-total=1024   free mem=918576864  used  
=330080544
LUCENE: offset=0  total=0  len-total=883   free mem=918576864  used =330080544
LUCENE: offset=0  total=0  len-total=1024   free mem=918576864  used  
=330080544
LUCENE: offset=0  total=0  len-total=1024   free mem=918576864  used  
=330080544
LUCENE: offset=0  total=0  len-total=1024   free mem=918576864  used  
=330080544
LUCENE: offset=0  total=0  len-total=1024   free mem=918576864  used  
=330080544
LUCENE: offset=0  total=0  len-total=1024   free mem=918576864  used  
=330080544
LUCENE: offset=0  total=0  len-total=1024   free mem=918576864  used  
=330080544
LUCENE: offset=0  total=0  len-total=1024   free mem=918576864  used  
=330080544
LUCENE: offset=0  total=0  len-total=1024   free mem=918576864  used  
=330080544
LUCENE: offset=0  total=0  len-total=209000000   free mem=631122912   
used =617534496
LUCENE: offset=209000000  total=0  len-total=20900000   free  
mem=631122912  used =617534496
LUCENE: offset=229900000  total=0  len-total=20900000   free  
mem=631122912  used =617534496
LUCENE: offset=250800000  total=0  len-total=20900000   free  
mem=631122912  used =617534496
...
completes successfully!


The method called via Hibernate Search:

...
LUCENE: offset=0  total=0  len-total=401   free mem=924185480  used =334892152
LUCENE: offset=0  total=0  len-total=1024   free mem=924185480  used  
=334892152
LUCENE: offset=0  total=0  len-total=883   free mem=924185480  used =334892152
LUCENE: offset=0  total=0  len-total=1024   free mem=924185480  used  
=334892152
LUCENE: offset=0  total=0  len-total=1024   free mem=924185480  used  
=334892152
LUCENE: offset=0  total=0  len-total=1024   free mem=924185480  used  
=334892152
LUCENE: offset=0  total=0  len-total=1024   free mem=924185480  used  
=334892152
LUCENE: offset=0  total=0  len-total=1024   free mem=924185480  used  
=334892152
LUCENE: offset=0  total=0  len-total=1024   free mem=924185480  used  
=334892152
LUCENE: offset=0  total=0  len-total=1024   free mem=924185480  used  
=334892152
LUCENE: offset=0  total=0  len-total=1024   free mem=924185480  used  
=334892152
LUCENE: offset=0  total=0  len-total=209000000   free mem=636731528   
used =622346104
Exception in thread "main" java.lang.OutOfMemoryError
         at java.io.RandomAccessFile.readBytes(Native Method)
         at java.io.RandomAccessFile.read(Unknown Source)
         at  
org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:599)
... fails with exception!


Note that the HS version has slightly more free memory because I ran  
it with -Xms1210M as opposed to -Xms1200M for the stand-alone to  
offset any memory used by HS when it starts up.

As you can see, these are identical for all practical purposes.  So  
what gives?

I'm stumped, so any suggestions appreciated.

Thanks.


Quoting Michael McCandless <lucene@mikemccandless.com>:

>
> Unfortunately, I'm not familiar with exactly what Hibernate search does
> with the Lucene APIs.
>
> It must be doing something beyond what your standalone Lucene test case does.
>
> Maybe ask this question on the Hibernate list?
>
> Mike



Mime
View raw message