lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Dunn <james_h_d...@yahoo.com>
Subject Re: Running OutOfMemory while optimizing and searching
Date Fri, 02 Jul 2004 17:29:13 GMT
Ah yes, I don't think I made that clear enough.  From
Mark's original post, I believe he mentioned that he
used seperate readers for each simultaneous query.

His other issue was that he was getting an OOM during
an optimize, even when he set the JVM heap to 2GB.  He
said his index was about 10.5GB spread over ~7000
files on Linux.  

My guess is that OOM might actually be a "too many
open files" error.  I have seen that type of error
being reported by the JVM as an OutOfMemory error on
Linux before.  I had the same problem but once I
switched to the new Lucene compound file format, I
haven't had that problem since.  

Mark, have you tried switching to the compound file
format?  

Jim




--- Doug Cutting <cutting@apache.org> wrote:
>  > What do your queries look like?  The memory
> required
>  > for a query can be computed by the following
> equation:
>  >
>  > 1 Byte * Number of fields in your query * Number
> of
>  > docs in your index
>  >
>  > So if your query searches on all 50 fields of
> your 3.5
>  > Million document index then each search would
> take
>  > about 175MB.  If your 3-4 searches run
> concurrently
>  > then that's about 525MB to 700MB chewed up at
> once.
> 
> That's not quite right.  If you use the same
> IndexSearcher (or 
> IndexReader) for all of the searches, then only
> 175MB are used.  The 
> arrays in question (the norms) are read-only and can
> be shared by all 
> searches.
> 
> In general, the amount of memory required is:
> 
> 1 byte * Number of searchable fields in your index *
> Number of docs in 
> your index
> 
> plus
> 
> 1k bytes * number of terms in query
> 
> plus
> 
> 1k bytes * number of phrase terms in query
> 
> The latter are for i/o buffers.  There are a few
> other things, but these 
> are the major ones.
> 
> Doug
> 
> 
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
> 
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message