lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@apache.org>
Subject Re: Running OutOfMemory while optimizing and searching
Date Fri, 17 Sep 2004 16:55:50 GMT
John Z wrote:
> We have indexes of around 1 million docs and around 25 searchable fields.
> We noticed that without any searches performed on the indexes, on startup, the memory
taken up by the searcher is roughly 7 times the .tii file size. The .tii file is read into
memory as per the code. Our .tii files are around 8-10 MB in size and our startup memory foot
print is around 60-70 MB.
>  
> Then when we start doing our searches, the memory goes up, depending on the fields we
search on. We are noticing that if we start searching on new fields, the memory kind of goes
up. 
>  
> Doug, 
>  
> Your calculation below on what is taken up by the searcher, does it take into account
the .tii file being read into memory  or am I not making any sense ? 
>  
> 1 byte * Number of searchable fields in your index * Number of docs in 
> your index
> plus
> 1k bytes * number of terms in query
> plus
> 1k bytes * number of phrase terms in query

You make perfect sense.  The formula above does not include the .tii. 
My mistake: I forgot that.  By default, every 128th Term in the index is 
read into memory, to permit random access to terms.  These are stored in 
the .tii file, compressed.  So it is not surprising that they require 7x 
the size of the .tii file in memory.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message