lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Keegan" <peterlkee...@gmail.com>
Subject Re: Custom sorting - memory leaks
Date Mon, 21 Aug 2006 16:54:27 GMT
Aleksey,

My app has precise and predictable control over when the index
reader/searcher gets refreshed. When this occurs, the sorting values for all
the docs in the new index are generated by the app and cached. Each query
thread has it's own ScoreDocComparator which contains a reference to the
sorting values and is reused with different 'point' values on each query.
So, Lucene only caches one instance of the ScoreDocComparator per query
thread.

Peter



On 8/21/06, Aleksey Serba <aserba@gmail.com> wrote:
>
> Peter,
>
> As i understand correctly, implementing the equals and hashCode
> methods let us avoid caching the same sorting values, i.e. if we have
> a few points to calc distance from then we can implement equals and
> hashCode methods based on point value and we'll have one cached
> comparator per different center point value.
>
> However, if every new search we have different center point value then
> we still in enormous memory usage problem.
>
> Aleksey
>
> On 8/21/06, Peter Keegan <peterlkeegan@gmail.com> wrote:
> > I avoided the cache by implementing the equals and hashCode methods
> > appropriately in my custom ScoreDocComparator as described here:
> > http://www.lucenebook.com/blog/errata/2006/03/01/memory_leak.html
> >
> > Peter
> >
> > On 8/21/06, Aleksey Serba <aserba@gmail.com> wrote:
> > >
> > > Hi Chris,
> > >
> > > On 5/17/06, Peter Keegan <peterlkeegan[at]gmail.com> wrote:
> > > > Suppose I have a custom sorting 'DocScoreComparator' for computing
> > > distances
> > > > on each search hit from a specified coordinate (similar to the
> > > > DistanceComparatorSource example in LIA). Assume that the 'specified
> > > > coordinate' is different for each query. This means a new custom
> > > comparator
> > > > must be created for each query, which is ok. However, Lucene caches
> the
> > > > comparator even though it will never be reused. This could result in
> > > heavy
> > > > memory usage if many queries are performed before the IndexReader is
> > > > updated.
> > >
> > > Lets suppose another custom sortings - random, using dynamically
> > > changed values from database, etc. Everytime lucene caches _useless_
> > > custom comparators and application developer has no any control on it.
> > >
> > > Also, I have no evidence to close IndexReader if my index is static.
> > >
> > > Aleksey
> > >
> > > On 8/18/06, Chris Hostetter <hossman_lucene@fucit.org> wrote:
> > > >
> > > > : You can reproduce OutOfMemory easily. I've attach test files -
> this is
> > > > : altered DistanceSortingTest example from LIA book. Also you can
> > > > : profile it and see caching of distances arrays.
> > > >
> > > > An OutOfMemory error is differnet from a memory leak.  Sorting with
> a
> > > > custom Comparator does in fact use a lot of memory -- and if your
> Heap
> > > > size is not big enough you may get an OutOfMemory -- but that
> doesn't
> > > mean
> > > > you have a memory leak.  A Memory leak is a problem where memory is
> > > > allocated for objects, but not freed up once those objects are no
> longer
> > > > of use -- just becuase you aren't using those objects, doesn't mean
> they
> > > > aren't "of use" and it doesn't mean they won't be freed up once some
> > > > futher action is taken (ie: closing of an IndexReader)
> > > >
> > > > Are you sure you've found an actual memory leak? ... ie: do you have
> a
> > > > demonstratable test case that shows heap usage growing without bound
> > > over
> > > > time?   If so then seeing *that* code would be helpful to figure out
> if
> > > > there is in fact a memory leak in Lucene, or if (perhaps) there is a
> > > > mistaken assumption in your code (ie; not closing an IndexReader)
> > > >
> > > >
> > > >
> > > >
> > > > -Hoss
> > > >
> > > >
> > > >
> ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > > > For additional commands, e-mail: java-dev-help@lucene.apache.org
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-dev-help@lucene.apache.org
> > >
> > >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message