lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-887) Interruptible segment merges
Date Fri, 01 Jun 2007 09:14:16 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12500634
] 

Michael McCandless commented on LUCENE-887:
-------------------------------------------

> > This actually means you can potentially have too many "level 0" (just
> > flushed) segments in the index but that should not be a big deal since
> > the next merge would clean it up. And it should be rare.
> 
> Yes, unless another shutdown request comes while the first merge
> after restarting the system is happening (which should be very
> unlikely), this will be cleaned up. Also, once the system is up
> again the IndexWriter will delete left over file fragments from an
> aborted merge.

Actually the leftover files from the aborted merge *should* be deleted
in the finally clause in mergeSegments, on hitting the original
IndexWriterInterruptedException.  Ie once shutdown is complete, your
index should have none of the partially written files left, I think?


> > Also I think it's possible that the addDocument() call from another
> > thread will hit the IndexWriterInterruptException, right? So those
> > other threads should catch this and ignore it (since their doc was in
> > fact succesfully added and only the followon merge was interrupted)?
> 
> Hmm I'm not sure if I understand this. I catch the
> IndexWriterInterruptException in addDocument() and in the catch
> block flushAfterInterrupt() is called which clears the interrupt
> flag. So IndexWriterInterruptException shouldn't be thrown again and
> addDocument() should just return normally? Or am I missing
> something. Could you give an example?

Woops -- sorry, I missed that you indeed catch the
IndexWriterInterruptException coming out of all addDocument calls, so
I think that will work correctly and the caller will never see the
exception.  Good!

Hmm, here's another possible issue .. thread safety sure is difficult.

Say there are 2 addDocument calls in-flight, and the first one tries
to flush buffered docs but hits the IndexWRiterInterruptException.
You catch this, clear the interrupt flag but before you then call
flushRAMSegments(false) the other addDocument thread get scheduled and
enters the synchronized block in addDocument and then kicks off a
flush with a merge.  I think at this point the merge would happily run
to completion because the interrupt flag got cleared?

> Interruptible segment merges
> ----------------------------
>
>                 Key: LUCENE-887
>                 URL: https://issues.apache.org/jira/browse/LUCENE-887
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index
>            Reporter: Michael Busch
>            Priority: Minor
>             Fix For: 2.2
>
>         Attachments: ExtendedIndexWriter.java
>
>
> Adds the ability to IndexWriter to interrupt an ongoing merge. This might be necessary
when Lucene is e. g. running as a service and has to stop indexing within a certain period
of time due to a shutdown request.
> A solution would be to add a new method shutdown() to IndexWriter which satisfies the
following two requirements:
> - if a merge is happening, abort it
> - flush the buffered docs but do not trigger a merge 
> See also discussions about this feature on java-dev:
> http://www.gossamer-threads.com/lists/lucene/java-dev/49008

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message