lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Keegan" <peterlkee...@gmail.com>
Subject Re: Custom sorting - memory leaks
Date Mon, 21 Aug 2006 17:09:14 GMT
I forgot to mention that the query threads are created only once.
Peter

On 8/21/06, Peter Keegan <peterlkeegan@gmail.com> wrote:
>
> Aleksey,
>
> My app has precise and predictable control over when the index
> reader/searcher gets refreshed. When this occurs, the sorting values for all
> the docs in the new index are generated by the app and cached. Each query
> thread has it's own ScoreDocComparator which contains a reference to the
> sorting values and is reused with different 'point' values on each query.
> So, Lucene only caches one instance of the ScoreDocComparator per query
> thread.
>
>
> Peter
>
>
>
> On 8/21/06, Aleksey Serba <aserba@gmail.com> wrote:
> >
> > Peter,
> >
> > As i understand correctly, implementing the equals and hashCode
> > methods let us avoid caching the same sorting values, i.e. if we have
> > a few points to calc distance from then we can implement equals and
> > hashCode methods based on point value and we'll have one cached
> > comparator per different center point value.
> >
> > However, if every new search we have different center point value then
> > we still in enormous memory usage problem.
> >
> > Aleksey
> >
> > On 8/21/06, Peter Keegan <peterlkeegan@gmail.com> wrote:
> > > I avoided the cache by implementing the equals and hashCode methods
> > > appropriately in my custom ScoreDocComparator as described here:
> > > http://www.lucenebook.com/blog/errata/2006/03/01/memory_leak.html
> > >
> > > Peter
> > >
> > > On 8/21/06, Aleksey Serba < aserba@gmail.com> wrote:
> > > >
> > > > Hi Chris,
> > > >
> > > > On 5/17/06, Peter Keegan <peterlkeegan[at]gmail.com> wrote:
> > > > > Suppose I have a custom sorting 'DocScoreComparator' for computing
> >
> > > > distances
> > > > > on each search hit from a specified coordinate (similar to the
> > > > > DistanceComparatorSource example in LIA). Assume that the
> > 'specified
> > > > > coordinate' is different for each query. This means a new custom
> > > > comparator
> > > > > must be created for each query, which is ok. However, Lucene
> > caches the
> > > > > comparator even though it will never be reused. This could result
> > in
> > > > heavy
> > > > > memory usage if many queries are performed before the IndexReader
> > is
> > > > > updated.
> > > >
> > > > Lets suppose another custom sortings - random, using dynamically
> > > > changed values from database, etc. Everytime lucene caches _useless_
> >
> > > > custom comparators and application developer has no any control on
> > it.
> > > >
> > > > Also, I have no evidence to close IndexReader if my index is static.
> > > >
> > > > Aleksey
> > > >
> > > > On 8/18/06, Chris Hostetter <hossman_lucene@fucit.org> wrote:
> > > > >
> > > > > : You can reproduce OutOfMemory easily. I've attach test files -
> > this is
> > > > > : altered DistanceSortingTest example from LIA book. Also you can
> > > > > : profile it and see caching of distances arrays.
> > > > >
> > > > > An OutOfMemory error is differnet from a memory leak.  Sorting
> > with a
> > > > > custom Comparator does in fact use a lot of memory -- and if your
> > Heap
> > > > > size is not big enough you may get an OutOfMemory -- but that
> > doesn't
> > > > mean
> > > > > you have a memory leak.  A Memory leak is a problem where memory
> > is
> > > > > allocated for objects, but not freed up once those objects are no
> > longer
> > > > > of use -- just becuase you aren't using those objects, doesn't
> > mean they
> > > > > aren't "of use" and it doesn't mean they won't be freed up once
> > some
> > > > > futher action is taken (ie: closing of an IndexReader)
> > > > >
> > > > > Are you sure you've found an actual memory leak? ... ie: do you
> > have a
> > > > > demonstratable test case that shows heap usage growing without
> > bound
> > > > over
> > > > > time?   If so then seeing *that* code would be helpful to figure
> > out if
> > > > > there is in fact a memory leak in Lucene, or if (perhaps) there is
> > a
> > > > > mistaken assumption in your code (ie; not closing an IndexReader)
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > -Hoss
> > > > >
> > > > >
> > > > >
> > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > > > > For additional commands, e-mail: java-dev-help@lucene.apache.org
> > > > >
> > > > >
> > > >
> > > >
> > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > > > For additional commands, e-mail: java-dev-help@lucene.apache.org
> > > >
> > > >
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message