lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] Updated: (LUCENE-1304) Memory Leak when using Custom Sort (i.e., DistanceSortSource) of LocalLucene with Lucene
Date Fri, 27 Mar 2009 20:55:50 GMT

     [ https://issues.apache.org/jira/browse/LUCENE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Michael McCandless updated LUCENE-1304:
---------------------------------------

    Fix Version/s: 2.9

> Memory Leak when using Custom Sort (i.e., DistanceSortSource) of LocalLucene with Lucene
> ----------------------------------------------------------------------------------------
>
>                 Key: LUCENE-1304
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1304
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Search
>    Affects Versions: 2.3
>         Environment: Windows/JDK 1.6
>            Reporter: Ethan Tao
>             Fix For: 2.9
>
>         Attachments: LUCENE-1304.patch
>
>
> We had the memory leak issue when using DistanceSortSource of LocalLucene for repeated
query/search. In about 450 queries, we are experiencing out of memory error. After dig in
the code, we found the problem source is coming from Lucene package, the way how it handles
"custom" type comparator. Lucene internally caches all created comparators. In the case of
query using LocalLucene, we create new comparator for every search due to different lon/lat
and query terms. This causes major memory leak as the cached comparators are also holding
memory for other large objects (e.g., bit sets). The solution we came up with: ( the proposed
change from Lucene is 1 and 3 below)
> 1.	In Lucene package, create new file SortComparatorSourceUncacheable.java:
> package org.apache.lucene.search;
> import org.apache.lucene.index.IndexReader;
> import java.io.IOException;
> import java.io.Serializable;
> public interface SortComparatorSourceUncacheable extends Serializable {
> }
> 2.	Have your custom sort class to implement the interface
> public class LocalSortSource extends DistanceSortSource implements SortComparatorSourceUncacheable
{
> ...
> }
> 3.	Modify Lucene's FieldSorterHitQueue.java to bypass caching for custom sort comparator:
> Index: FieldSortedHitQueue.java
> ===================================================================
> --- FieldSortedHitQueue.java     (revision 654583)
> +++ FieldSortedHitQueue.java  (working copy)
> @@ -53,7 +53,12 @@
>      this.fields = new SortField[n];
>      for (int i=0; i<n; ++i) {
>        String fieldname = fields[i].getField();
> -      comparators[i] = getCachedComparator (reader, fieldname, fields[i].getType(),
fields[i].getLocale(), fields[i].getFactory());
> +
> +      if(fields[i].getFactory() instanceof SortComparatorSourceUncacheable) { // no
caching to avoid memory leak
> +        comparators[i] = getComparator (reader, fieldname, fields[i].getType(), fields[i].getLocale(),
fields[i].getFactory());
> +      } else {
> +        comparators[i] = getCachedComparator (reader, fieldname, fields[i].getType(),
fields[i].getLocale(), fields[i].getFactory());
> +      }
>        
>        if (comparators[i].sortType() == SortField.STRING) {
>                   this.fields[i] = new SortField (fieldname, fields[i].getLocale(), fields[i].getReverse());
> @@ -157,7 +162,18 @@
>    SortField[] getFields() {
>      return fields;
>    }
> -  
> +
> +  static ScoreDocComparator getComparator (IndexReader reader, String field, int type,
Locale locale, SortComparatorSource factory)
> +    throws IOException {
> +      if (type == SortField.DOC) return ScoreDocComparator.INDEXORDER;
> +      if (type == SortField.SCORE) return ScoreDocComparator.RELEVANCE;
> +      FieldCacheImpl.Entry entry = (factory != null)
> +        ? new FieldCacheImpl.Entry (field, factory)
> +        : new FieldCacheImpl.Entry (field, type, locale);
> +      return (ScoreDocComparator)Comparators.createValue(reader, entry);
> +    }
> +
> +
> Otis suggests that I put this in Jira. I 'll attach a patch shortly for review. 
> -Ethan

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message