lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Willnauer (JIRA)" <>
Subject [jira] [Commented] (LUCENE-3348) IndexWriter applies wrong deletes during concurrent flush-all
Date Mon, 01 Aug 2011 19:31:10 GMT


Simon Willnauer commented on LUCENE-3348:

mike I can not reproduce this failure.. what exactly is failing there? maybe you can put the
output in a text file and attache it?

Regarding the latest patch, I think we can call DWFlushControl#addFlushableState() from DWFlushControl#markForFullFlush()
and use a global list to collect the DWPT for the full flush. 

I think we should move the getAndLock call into DWFlushControl something like DWFlushControl#obtainAndLock(),
this would allow us to make the check and the DWFlushControl#addFlushableState() method private
to DWFC. Further we can also simplify the deleteQueue check a little since we already obtained
a ThreadState we don't need to unlock the state again after calling addFlushableState(), something
like this:

ThreadState obtainAndLock() {
    final ThreadState perThread = perThreadPool.getAndLock(Thread
        .currentThread(), documentsWriter);
    if (perThread.isActive()
        && perThread.perThread.deleteQueue != documentsWriter.deleteQueue) {
      // There is a flush-all in process and this DWPT is
      // now stale -- enroll it for flush and try for
      // another DWPT:
    return perThread;

Eventually we are spending too much time in full flush since we lock all ThreadStates at least
once while some indexing threads might have already helped out with swapping out DWPT instances.
I think we can collect already swapped out ThreadStates during a full flush and only check
the ones that have not been processed? 

> IndexWriter applies wrong deletes during concurrent flush-all
> -------------------------------------------------------------
>                 Key: LUCENE-3348
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: core/index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.4, 4.0
>         Attachments: LUCENE-3348.patch, LUCENE-3348.patch, LUCENE-3348.patch
> Yonik uncovered this with the TestRealTimeGet test: if a flush-all is
> underway, it is possible for an incoming update to pick a DWPT that is
> stale, ie, not yet pulled/marked for flushing, yet the DW has cutover
> to a new deletes queue.  If this happens, and the deleted term was
> also updated in one of the non-stale DWPTs, then the wrong document is
> deleted and the test fails by detecting the wrong value.
> There's a 2nd failure mode that I haven't figured out yet, whereby 2
> docs are returned when searching by id (there should only ever be 1
> doc since the test uses updateDocument which is atomic wrt
> commit/reopen).
> Yonik verified the test passes pre-DWPT, so my guess is (but I
> have yet to verify) this test also passes on 3.x.  I'll backport
> the test to 3.x to be sure.

This message is automatically generated by JIRA.
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message