lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Willnauer (JIRA)" <>
Subject [jira] [Updated] (LUCENE-5006) Simplify / understand IndexWriter/DocumentsWriter synchronization
Date Wed, 19 Jun 2013 08:26:21 GMT


Simon Willnauer updated LUCENE-5006:

    Attachment: LUCENE-5006.patch

Here is a cleaned-up version of the patch.

I removed the accidentally added (leftover) int[] from BytesRefHash that was indeed unintended.

I also removed all the leftovers like forcePurge and applyDeletes flags they were still in
there from a previous iteration without the Queue. I changed _maybeMerge_ to _hasEvents_ consistently.

The changes in DWPT and DWPTThreadPool are mainly due to the fact that I move the creation
of DWPT into DW and out of the ThreadPool. The ThreadPool only maintains the ThreadState instances
but is not responsible for creating the actual DWPT. DWPT is now not "reuseable" anymore,
yet we never really reused them but if they were initialized and we did a full flush we kept
using them with a new DeleteQueue which is gone now. This is nice since DWPT is now solely
initialized in its Ctor. This includes the segment name which we obtain from IW when the DWPT
is created. This remains the only place where we sync on IW which is done in updateDocument
right now. 

I think this patch is a step into the right direction making this simpler, at the end of the
day I'd want to change the lifetime of a DW to be a single flush and replace the entire DW
once we flush or reopen. This would make a lot of logic much simpler but I don't want to make
this big change at once so maybe we should work to get the current patch into trunk and let
it bake in a bit.
> Simplify / understand IndexWriter/DocumentsWriter synchronization
> -----------------------------------------------------------------
>                 Key: LUCENE-5006
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Bug
>            Reporter: Michael McCandless
>            Assignee: Simon Willnauer
>         Attachments: LUCENE-5006.patch, LUCENE-5006.patch
> The concurrency in IW/DW/BD is terrifying: there are many locks involved, not just intrinsic
locks but IW also has fullFlushLock, commitLock, and there are no clear rules about lock order
to avoid deadlocks like LUCENE-5002.
> We have to somehow simplify this, and define the allowed concurrent behavior eg when
an app calls deleteAll while other threads are indexing.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message