lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From eks dev <eks...@yahoo.co.uk>
Subject Re: Out of memory - CachingWrappperFilter and multiple threads
Date Tue, 19 Feb 2008 11:58:34 GMT
hi Mark, 

just out of curiosity, do you know the distribution of set bits  in these terms you have tried
to cache? 
maybe this simple tip could help.
If you are lucky like we were, such terms typically used for filters are good candidates to
be used to sort your index before indexing (once in a while) and then with some sort of IntervalDocIdSet
you can reduce memory requirements dramatically. 

 

----- Original Message ----
From: markharw00d <markharw00d@yahoo.co.uk>
To: java-dev@lucene.apache.org
Sent: Tuesday, 19 February, 2008 9:20:02 AM
Subject: Re: Out of memory - CachingWrappperFilter and multiple threads

I now think the main issue here is that a busy JVM gets into trouble 
trying to find large free blocks of memory for large bitsets.
In my index of 64 million documents, ~8meg of contiguous free memory 
must be found for each bitset allocated. The terms I was trying to cache 
had 14 million entries so the new DocIdSet alternatives for bitsets 
probably fare no better.

The JVM (Sun 1..5) doesn't seem to deal with these allocations well. 
Perhaps there's an obscure JVM option I can set to reserve a section of 
RAM for large allocations.
However, I wonder if we should help the JVM out a little here by having 
pre-allocated pools of BitsSets/OpenBitSets that can be reserved and 
reused by the application. This would imply a change to filter classes 
so instead of constructing BitSets/OpenBitsets directly they get them 
from a pool instead.

Thoughts?


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org






      __________________________________________________________
Sent from Yahoo! Mail - a smarter inbox http://uk.mail.yahoo.com



---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message