lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steven Parkes (JIRA)" <>
Subject [jira] Updated: (LUCENE-870) add concurrent merge policy
Date Mon, 20 Aug 2007 20:13:30 GMT


Steven Parkes updated LUCENE-870:

    Attachment: CMP.patch.txt

Mike expressed interest in pursuing this with an alternative strargey, so I thought I'd give
a "work in progress" snapshot of the way I'd be going.

This code doesn't work, but it has some ideas, so it's only of interest to people who really
want to make suggestions on how to do the parallelization.

Overall, the idea of generating new threads for non-conflicting primitive merges seems okay.
Need to make sure you don't overload the i/o system and that throttling code isn't in there.
A couple of things things off the top that I haven't worked through yet:

My current thinking is that when you are not going to do a merge serially, you need to copy
the segmentInfo objects that you will be using. It may be possible to do this with a lock,
but that gets harry. Ther'es also state in the SegmentInfo objescts themselves, like docStoreIsCompoundFile
that can get changed on the fly.

flushDocStore is challenging to parallelize. It's synchronized now, but you probably would
rather it not be? It's complicated by the fact that doc stores are shared by multiple segments
and so non-conflicting merges may stll share doc stores.

> add concurrent merge policy
> ---------------------------
>                 Key: LUCENE-870
>                 URL:
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index
>            Reporter: Steven Parkes
>            Assignee: Steven Parkes
>         Attachments: CMP.patch.txt, concurrentMerge.patch
> Provide the ability to handle merges in one or more concurrent threads, i.e., concurrent
with other IndexWriter operations.
> I'm factoring the code from LUCENE-847 for this.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message