lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yonik Seeley <yo...@lucidimagination.com>
Subject Re: memory leak with CustomComparatorSource class variables
Date Sat, 13 Jun 2009 13:39:03 GMT
When implementing your own, it also helps to look at the existing
implementations in the FieldComparator class:

http://svn.apache.org/viewvc/lucene/java/trunk/src/java/org/apache/lucene/search/FieldComparator.java?revision=764551

-Yonik
http://www.lucidimagination.com


On Sat, Jun 13, 2009 at 9:30 AM, Marc Sturlese<marc.sturlese@gmail.com> wrote:
>
> Thanks Mike, really useful info. I have dowloaded the latest Lucene 2.9-dev
> to test the implementation of a FieldComparatorSource but the API
> documentation doesn't seem to be availabe.
>
> I can access to the class MissingStringLastComparatorSource:
> http://lucene.apache.org/solr/api/org/apache/solr/search/MissingStringLastComparatorSource.html
> From there I try to link to org.apache.lucene.search.FieldComparatorSource
> but get a 404 error.
> Any idea how can I get access to that documentation?
>
> Thanks in advance!
>
>
> Michael McCandless-2 wrote:
>>
>> On Fri, Jun 12, 2009 at 6:09 PM, Marc Sturlese<marc.sturlese@gmail.com>
>> wrote:
>>
>>> I have noticed I am experiencing sort of a memory leak with a
>>> CustomComparatorSource (wich implements SortComparatorSource).
>>> I have a HashMap declared as variable of class in CustomComparatorSource:
>>
>> This is unfortunately a known and rather horrific trap, in Lucene.
>>
>> Lucene's field sorting implementation (FieldSortedHitQueue) caches the
>> comparators use during sorting (in its static package private
>> Comparators field).  They are weakly keyed by IndexReader, so that
>> when the IndexReader is closed, the cache entries are cleared.  When
>> sorting by field this is normally OK since we hold a single entry for
>> that field.
>>
>> But when you provide a SortComparatorSource, it's included in the
>> cache key.  So, if you don't implement hashCode/equals (correctly), or
>> using a singleton or restricted set of instances (say), then suddenly
>> every new instance of your SortComparatorSource will enter the cache
>> and not be cleared until you close that reader.  It easily results in
>> a catastrophic, extremely unexpected memory leak.
>>
>> Lucene 2.9 has fixed this, as a side effect of the move to per-segment
>> field sorting.  SortComparatorSource is replaced by
>> FieldComparatorSource, and this caching of comparators is no longer
>> done.  Another reason to get 2.9 out sooner rather than later!
>>
>> Mike
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/memory-leak-with-CustomComparatorSource-class-variables-tp24006806p24012496.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message