lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From robert engels <reng...@ix.netcom.com>
Subject Re: Out of memory - CachingWrappperFilter and multiple threads
Date Mon, 18 Feb 2008 15:14:05 GMT
Even though you only have two threads... how many different  
IndexReader instances do you have?

Also, if they are opened/closed/released, and you are running at 100  
% cpu, you may be running into a finalization problem - the instances/ 
references cannot be GC'd due to finalization, thus the OOM.

On Feb 18, 2008, at 6:50 AM, mark harwood wrote:

> I'm chasing down a bug in my application where multiple threads  
> were readingand caching the same filter (same very common term, big  
> index) and causedan Out of Memory exception when I would expect  
> there to be plenty ofmemory to spare.
> There's a number of layers to this app to investigate (I was using  
> theXMLQueryParser and the CachedFilter tag too) but  
> CachingWrapperFilterunderpins all this stuff and I was led to this  
> code in it...
>
>   public BitSet bits(IndexReader reader) throws IOException {
>     if (cache == null) {
>       cache = new WeakHashMap();
>     }
>
>     synchronized (cache) {  // check cache
>       BitSet cached = (BitSet) cache.get(reader);
>       if (cached != null) {
>         return cached;
>       }
>     }
>
>     final BitSet bits = filter.bits(reader);
>
>     synchronized (cache) {  // update cache
>       cache.put(reader, bits);
>     }
>
>     return bits;
>   }
>
>
> The first observation is - why the use of"final" for the variable  
> "bits" ?  Would there be anyside-effects to this?
>
> Perhaps more worryingly I can see that multiple threads asking for  
> the same bitset simultaneously arelikely to unnecessarily read the  
> same data from the same reader (butultimately only one bitset  
> should end up cached). My app only had 2 simultaneous threads on  
> the same reader so I don't see how that accounts for the large  
> memory bloat I saw. In a high traffic environment though, I can see  
> multiple requests for a popular term getting bottle-necked here  
> creating the same bitset and causing an OOM error. It looks like  
> this multiple-load scenario could/should be avoided with some  
> careful synchronisation.
>
> Unfortunately I've been unable to reproduce my OOM problem outside  
> of the live environment so can't fully pinpoint my particular issue  
> or the solution just yet.
>
> Thoughts?
> Mark
>
>
>
>
>       __________________________________________________________
> Sent from Yahoo! Mail - a smarter inbox http://uk.mail.yahoo.com
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message