lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aleksey Serba" <ase...@gmail.com>
Subject Re: Custom sorting - memory leaks
Date Mon, 21 Aug 2006 14:24:33 GMT
Peter,

As i understand correctly, implementing the equals and hashCode
methods let us avoid caching the same sorting values, i.e. if we have
a few points to calc distance from then we can implement equals and
hashCode methods based on point value and we'll have one cached
comparator per different center point value.

However, if every new search we have different center point value then
we still in enormous memory usage problem.

Aleksey

On 8/21/06, Peter Keegan <peterlkeegan@gmail.com> wrote:
> I avoided the cache by implementing the equals and hashCode methods
> appropriately in my custom ScoreDocComparator as described here:
> http://www.lucenebook.com/blog/errata/2006/03/01/memory_leak.html
>
> Peter
>
> On 8/21/06, Aleksey Serba <aserba@gmail.com> wrote:
> >
> > Hi Chris,
> >
> > On 5/17/06, Peter Keegan <peterlkeegan[at]gmail.com> wrote:
> > > Suppose I have a custom sorting 'DocScoreComparator' for computing
> > distances
> > > on each search hit from a specified coordinate (similar to the
> > > DistanceComparatorSource example in LIA). Assume that the 'specified
> > > coordinate' is different for each query. This means a new custom
> > comparator
> > > must be created for each query, which is ok. However, Lucene caches the
> > > comparator even though it will never be reused. This could result in
> > heavy
> > > memory usage if many queries are performed before the IndexReader is
> > > updated.
> >
> > Lets suppose another custom sortings - random, using dynamically
> > changed values from database, etc. Everytime lucene caches _useless_
> > custom comparators and application developer has no any control on it.
> >
> > Also, I have no evidence to close IndexReader if my index is static.
> >
> > Aleksey
> >
> > On 8/18/06, Chris Hostetter <hossman_lucene@fucit.org> wrote:
> > >
> > > : You can reproduce OutOfMemory easily. I've attach test files - this is
> > > : altered DistanceSortingTest example from LIA book. Also you can
> > > : profile it and see caching of distances arrays.
> > >
> > > An OutOfMemory error is differnet from a memory leak.  Sorting with a
> > > custom Comparator does in fact use a lot of memory -- and if your Heap
> > > size is not big enough you may get an OutOfMemory -- but that doesn't
> > mean
> > > you have a memory leak.  A Memory leak is a problem where memory is
> > > allocated for objects, but not freed up once those objects are no longer
> > > of use -- just becuase you aren't using those objects, doesn't mean they
> > > aren't "of use" and it doesn't mean they won't be freed up once some
> > > futher action is taken (ie: closing of an IndexReader)
> > >
> > > Are you sure you've found an actual memory leak? ... ie: do you have a
> > > demonstratable test case that shows heap usage growing without bound
> > over
> > > time?   If so then seeing *that* code would be helpful to figure out if
> > > there is in fact a memory leak in Lucene, or if (perhaps) there is a
> > > mistaken assumption in your code (ie; not closing an IndexReader)
> > >
> > >
> > >
> > >
> > > -Hoss
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-dev-help@lucene.apache.org
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message