lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Keegan" <peterlkee...@gmail.com>
Subject Re: Custom sorting - memory leaks
Date Mon, 21 Aug 2006 13:51:54 GMT
I avoided the cache by implementing the equals and hashCode methods
appropriately in my custom ScoreDocComparator as described here:
http://www.lucenebook.com/blog/errata/2006/03/01/memory_leak.html

Peter

On 8/21/06, Aleksey Serba <aserba@gmail.com> wrote:
>
> Hi Chris,
>
> On 5/17/06, Peter Keegan <peterlkeegan[at]gmail.com> wrote:
> > Suppose I have a custom sorting 'DocScoreComparator' for computing
> distances
> > on each search hit from a specified coordinate (similar to the
> > DistanceComparatorSource example in LIA). Assume that the 'specified
> > coordinate' is different for each query. This means a new custom
> comparator
> > must be created for each query, which is ok. However, Lucene caches the
> > comparator even though it will never be reused. This could result in
> heavy
> > memory usage if many queries are performed before the IndexReader is
> > updated.
>
> Lets suppose another custom sortings - random, using dynamically
> changed values from database, etc. Everytime lucene caches _useless_
> custom comparators and application developer has no any control on it.
>
> Also, I have no evidence to close IndexReader if my index is static.
>
> Aleksey
>
> On 8/18/06, Chris Hostetter <hossman_lucene@fucit.org> wrote:
> >
> > : You can reproduce OutOfMemory easily. I've attach test files - this is
> > : altered DistanceSortingTest example from LIA book. Also you can
> > : profile it and see caching of distances arrays.
> >
> > An OutOfMemory error is differnet from a memory leak.  Sorting with a
> > custom Comparator does in fact use a lot of memory -- and if your Heap
> > size is not big enough you may get an OutOfMemory -- but that doesn't
> mean
> > you have a memory leak.  A Memory leak is a problem where memory is
> > allocated for objects, but not freed up once those objects are no longer
> > of use -- just becuase you aren't using those objects, doesn't mean they
> > aren't "of use" and it doesn't mean they won't be freed up once some
> > futher action is taken (ie: closing of an IndexReader)
> >
> > Are you sure you've found an actual memory leak? ... ie: do you have a
> > demonstratable test case that shows heap usage growing without bound
> over
> > time?   If so then seeing *that* code would be helpful to figure out if
> > there is in fact a memory leak in Lucene, or if (perhaps) there is a
> > mistaken assumption in your code (ie; not closing an IndexReader)
> >
> >
> >
> >
> > -Hoss
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message