lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yonik Seeley <ysee...@gmail.com>
Subject Re: optimized reopen?
Date Mon, 16 May 2005 18:50:09 GMT
Oops, this fell off lucene-dev... re-adding it.
-Yonik

On 5/16/05, Yonik Seeley <yseeley@gmail.com> wrote:
> In general, people shouldn't be relying on GC to clean up non-memory
> resources.  If you don't explicitly call close() on an IndexReader
> when you are done with it, you are likely to run out of file
> descriptors at some point.  Finalizers are a bad solution to clean up
> descriptors.
> 
> With non-shared readers, things would continue to work like normal.
> The constructor could set all the counts to 1, and close() could
> decrement and really do a close if it hits 0.
> 
> IndexReader.reopen() could then just increment the count of any reader
> it wished to share.
> 
> -Yonik
> 
> On 5/16/05, Robert Engels <rengels@ix.netcom.com> wrote:
> > Agreed, but ref counting can be a tricky business in a GC environment. It
> > may be better to use weak references, and let Java collect them when needed.
> > ?
> >
> > -----Original Message-----
> > From: Yonik Seeley [mailto:yseeley@gmail.com]
> > Sent: Monday, May 16, 2005 1:11 PM
> > To: rengels@ix.netcom.com
> > Subject: Re: optimized reopen?
> >
> > If the SegmentReaders can be shared, it looks like the only thing that
> > would need changing is a reference count on all IndexReaders to avoid
> > closing of a shared reader.
> >
> > -Yonik
> >
> > On 5/14/05, Robert Engels <rengels@ix.netcom.com> wrote:
> > > It appears so, as the SegmentReaders are read-only.
> > >
> > > -----Original Message-----
> > > From: Yonik Seeley [mailto:yseeley@gmail.com]
> > > Sent: Saturday, May 14, 2005 2:43 PM
> > > To: rengels@ix.netcom.com
> > > Subject: Re: optimized reopen?
> > >
> > > Our uses involve overlapping IndexReaders... one searcher is "warming
> > > up" while the other one is finishing queries.  Even after the new
> > > searcher switches to be the default one, the old one must be kept
> > > around until all queries using it finish.
> > >
> > > So can the segment readers be shared by multiple active index readers?
> > >
> > > -Yonik
> > >
> > > On 5/11/05, Robert Engels <rengels@ix.netcom.com> wrote:
> > > > I did an implementation of IndexReader.reopen() in my modified Lucene
> > lib.
> > > >
> > > > It is over 200% faster than closing and opening the index.
> > > >
> > > > I just made the semantics of IndexReader.reopen() be that the original
> > > > IndexReader is now closed, and can no longer be used (since I cannot
> > > close()
> > > > the original, otherwise the SegmentReaders will close, so I close the
> > > unused
> > > > SegmentReaders directly). In order to make this work I also needed to
> > make
> > > > commit() public, since the commit() needs to be performed prior to the
> > > > segment merging, and I needed to add a package method to MultiReader to
> > > get
> > > > the underlying subReaders.
> > > >
> > > > Although this may not be useful to many, in our case since we use a
> > single
> > > > shared reader and a single exclusive writer in a server process (with
> > > > external locking, so we can guarantee than when the IndexReader is
> > > "rolled",
> > > > there are no references to the previous reader.
> > > >
> > > > Robert
> > > >
> > > > -----Original Message-----
> > > > From: Yonik Seeley [mailto:yseeley@gmail.com]
> > > > Sent: Wednesday, May 11, 2005 4:50 PM
> > > > To: java-dev@lucene.apache.org; rengels@ix.netcom.com
> > > > Subject: Re: optimized reopen?
> > > >
> > > > Things are cached using an IndexReader as the key, so you would have
> > > > to be careful not to break the current behaviour (that an
> > > > IndexReader's view of an index doesn't change - deletes from that
> > > > specific reader aside).
> > > >
> > > > Maybe you could invoke reopen() on an existing IndexReader and it
> > > > would return a new IndexReader that shares sub-readers that haven't
> > > > changed?  But I don't think that sub-readers can be shared like this
> > > > right now... the difficulty may lie in deleted-docs.
> > > >
> > > > -Yonik
> > > >
> > > > On 5/11/05, Robert Engels <rengels@ix.netcom.com> wrote:
> > > > > Is there any way to optimize the closing & reopening of an Index?
> > > > >
> > > > > Since the IndexReader.open() opens a MultiReader is there are multiple
> > > > > segments, it seems a reopen() method could be implemented, which
> > detects
> > > > > which segments are the same as the current open index, and then passes
> > > > those
> > > > > SegementReaders to a new Multireader rather than creating new
> > > > SegmentReaders
> > > > > for the unmodified segments.
> > > > >
> > > > > Does this sounds feasible? Would it improve the performance?
> > > > >
> > > > > Thanks
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message