lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Busch (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-887) Interruptible segment merges
Date Fri, 01 Jun 2007 01:51:15 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12500550
] 

Michael Busch commented on LUCENE-887:
--------------------------------------

> This looks great to me!

Thanks for reviewing!

> So, if a shutdown request comes in then currently running addDocument
> calls are allowed to complete but if a new addDocument call tries to
> run it will hit an "IndexWriter already closed" IOException.  Once the
> in-flight addDocument calls finish you then flush the ram segments
> without allowing cascading merge.

Exactly.

> This actually means you can potentially have too many "level 0" (just
> flushed) segments in the index but that should not be a big deal since
> the next merge would clean it up.  And it should be rare.

Yes, unless another shutdown request comes while the first merge after
restarting the system is happening (which should be very unlikely), this
will be cleaned up. Also, once the system is up again the IndexWriter 
will delete left over file fragments from an aborted merge.

> In shutdown(), after you call waitForAddDocument(), why not call
> clearInterrupt before calling flushRamSegments?  Isn't the
> flushRamSegments() call guaranteed to hit the
> IndexWriterInterruptException if it's using an ExtendedFSDirectory and
> there are > 0 buffered docs?

Hmm I think I did it this way in case we aren't using an 
ExtendedFSDirectory, because then the flush would just succeed without 
an IndexWriterInterruptException and we safe an instanceof check here. 
But you are right, we can just call clearInterrupt, but only if 
(d instanceof ExtendedFSDirectory) == true. That's probably simpler. 
Thereafter it is safe to call close() because the buffer is empty, so 
the call of flushRamSegments in close() won't do anything.

> Also I think it's possible that the addDocument() call from another
> thread will hit the IndexWriterInterruptException, right?  So those
> other threads should catch this and ignore it (since their doc was in
> fact succesfully added and only the followon merge was interrupted)?

Hmm I'm not sure if I understand this. I catch the 
IndexWriterInterruptException in addDocument() and in the catch block
flushAfterInterrupt() is called which clears the interrupt flag. So
IndexWriterInterruptException shouldn't be thrown again and addDocument()
should just return normally? Or am I missing something. Could you give 
an example?

> Interruptible segment merges
> ----------------------------
>
>                 Key: LUCENE-887
>                 URL: https://issues.apache.org/jira/browse/LUCENE-887
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index
>            Reporter: Michael Busch
>            Priority: Minor
>             Fix For: 2.2
>
>         Attachments: ExtendedIndexWriter.java
>
>
> Adds the ability to IndexWriter to interrupt an ongoing merge. This might be necessary
when Lucene is e. g. running as a service and has to stop indexing within a certain period
of time due to a shutdown request.
> A solution would be to add a new method shutdown() to IndexWriter which satisfies the
following two requirements:
> - if a merge is happening, abort it
> - flush the buffered docs but do not trigger a merge 
> See also discussions about this feature on java-dev:
> http://www.gossamer-threads.com/lists/lucene/java-dev/49008

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message