lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Busch (JIRA)" <>
Subject [jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that write their own private segments
Date Mon, 29 Mar 2010 04:50:27 GMT


Michael Busch commented on LUCENE-2324:

However, in the apply deletes
method how would we know which doc to stop deleting at? How
would the seq id map to a DW's doc id?

We could have a global deletes-map that stores seqID -> DeleteAction.  DeleteAction either
contains a Term or a Query, and in addition an int "flushCount" (I'll explain in a bit what
flushCount is used for.)

Each DocumentsWriterPerThread would have a growing array that contains each seqID that "affected"
that DWPT, i.e. the seqIDs of *all* deletes, plus the seqIDs of the adds/updates performed
by that particular DWPT.  One bit of a seqID in that array can indicate if it's a delete or

When it's time to flush we sort the array by increasing seqID and then loop a single time
through it to find the seqIDs of all DeleteActions.  During the loop we count the number of
adds/updates to determine the number of docs the DeleteActions affect.  After applying the
deletes the DWPT makes a synchronized call to the global deletes-map and increments the flushCount
int for each applied DeleteAction.  If flushCount==numThreadStates (== number of DWPT instances)
the corresponding DeleteAction entry can be removed, because it was applied to all DWPT.

I think this should work?  Or is there a simpler solution?

> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>                 Key: LUCENE-2324
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael Busch
>            Assignee: Michael Busch
>            Priority: Minor
>             Fix For: 3.1
>         Attachments: LUCENE-2324.patch
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message