lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Florence" <mflore...@airsmail.com>
Subject RE: Running OutOfMemory while optimizing and searching
Date Fri, 02 Jul 2004 18:07:12 GMT
Thanks, Jim. I'm pretty sure I'm throwing OOM for real,
and not because I've run out of file handles. I can easily
recreate the latter condition, and it is always reported
accurately. I've also monitored the OOM as it occurs using
"top" and I can see memory usage climbing until it is
exhausted -- if you will excuse the pun!

I'm not familiar with the new compound file format. Where
can I look to find more information?

-- Mark

-----Original Message-----
From: James Dunn [mailto:james_h_dunn@yahoo.com]
Sent: Friday, July 02, 2004 01:29 pm
To: Lucene Users List
Subject: Re: Running OutOfMemory while optimizing and searching


Ah yes, I don't think I made that clear enough.  From
Mark's original post, I believe he mentioned that he
used seperate readers for each simultaneous query.

His other issue was that he was getting an OOM during
an optimize, even when he set the JVM heap to 2GB.  He
said his index was about 10.5GB spread over ~7000
files on Linux.  

My guess is that OOM might actually be a "too many
open files" error.  I have seen that type of error
being reported by the JVM as an OutOfMemory error on
Linux before.  I had the same problem but once I
switched to the new Lucene compound file format, I
haven't had that problem since.  

Mark, have you tried switching to the compound file
format?  

Jim




--- Doug Cutting <cutting@apache.org> wrote:
>  > What do your queries look like?  The memory
> required
>  > for a query can be computed by the following
> equation:
>  >
>  > 1 Byte * Number of fields in your query * Number
> of
>  > docs in your index
>  >
>  > So if your query searches on all 50 fields of
> your 3.5
>  > Million document index then each search would
> take
>  > about 175MB.  If your 3-4 searches run
> concurrently
>  > then that's about 525MB to 700MB chewed up at
> once.
> 
> That's not quite right.  If you use the same
> IndexSearcher (or 
> IndexReader) for all of the searches, then only
> 175MB are used.  The 
> arrays in question (the norms) are read-only and can
> be shared by all 
> searches.
> 
> In general, the amount of memory required is:
> 
> 1 byte * Number of searchable fields in your index *
> Number of docs in 
> your index
> 
> plus
> 
> 1k bytes * number of terms in query
> 
> plus
> 
> 1k bytes * number of phrase terms in query
> 
> The latter are for i/o buffers.  There are a few
> other things, but these 
> are the major ones.
> 
> Doug
> 
> 
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
> 
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message