lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless" <luc...@mikemccandless.com>
Subject RE: IndexWriter shutdown
Date Wed, 23 May 2007 21:19:19 GMT
"Doron Cohen" <DORONC@il.ibm.com> wrote:
>
> "Michael McCandless" wrote:
> 
> > That's correct.
> >
> > On seeing the "shutdown in progress" exception, the current "finally"
> > clause in mergeSegments would revert the internal state of the
> > IndexWriter to be consistent, ie, put back the segments that were in
> > the process of being merged into its segmentInfos.  It will also
> > remove any partially created but now unusable newly merged segments
> > files.
> >
> > If the application catches this exception and calls
> > IndexWriter.close(), then the state until just before the aborted
> > merge would be committed to the index.  If instead the application
> > catches the exception and does nothing, then the state of the index
> > reverts back to where it was when this IndexWriter instance was first
> > opened.
> >
> > So the semantics of autoCommit=false will be correctly enforced if any
> > exception (not just this new one) comes up through mergeSegments.
> 
> Great.
> 
> So my comment on Antony's "mini-optimize" scenario was
> partially wrong, because under autcCommit=true (which is
> the default), those sub-merges that completed before shutdown
> are not lost, only the last one, the one that was interrupted.

Right.
 
> Mmmm... I can see how autocommit=true works fine, because
> anything (auto)committed is already saved, and there
> is no need to write anything more.  But for autoCommit=false
> it is not clear to me how such further call to indexWriter.close()
> by the application can work - because a shutdown state is in
> effect, and any attempt to write/flush anything would just throw
> the same exception again...  or am I missing something?

Ahh, you are correct: the global/static shutdown state would prevent
any further writes, so if the IndexWriter.close() tried to write the
new segments_N, it would hit the same exception.

Maybe this isn't really a big deal?  Ie people who open an IndexWriter
with autoCommit=false should be prepared on shutdown to lose all that
had been done during the lifetime of that writer?  Presumably faced
with this you would just open a new writer exclusively to do the
optimize.  Though for the merging case, which you can't control (just
happens on certain addDocument(...) calls) that's harder because you
could then lose added documents.

Or, maybe, you have a way to "un-shutdown" and you call this before
calling close?  Or instead of "shutdown" it's more of a "interrupt the
merge if it's in progress" which then doesn't prevent further IO?
This is getting somewhat complex...

Maybe we should leave this out of the core, and instead implement as
[external] subclass of FSDirectory, until we can get a better handle
on it?

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message