lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From markharw00d <markharw...@yahoo.co.uk>
Subject Re: Out of memory - CachingWrappperFilter and multiple threads
Date Tue, 19 Feb 2008 08:20:02 GMT
I now think the main issue here is that a busy JVM gets into trouble 
trying to find large free blocks of memory for large bitsets.
In my index of 64 million documents, ~8meg of contiguous free memory 
must be found for each bitset allocated. The terms I was trying to cache 
had 14 million entries so the new DocIdSet alternatives for bitsets 
probably fare no better.

The JVM (Sun 1.5) doesn't seem to deal with these allocations well. 
Perhaps there's an obscure JVM option I can set to reserve a section of 
RAM for large allocations.
However, I wonder if we should help the JVM out a little here by having 
pre-allocated pools of BitsSets/OpenBitSets that can be reserved and 
reused by the application. This would imply a change to filter classes 
so instead of constructing BitSets/OpenBitsets directly they get them 
from a pool instead.

Thoughts?


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message