lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] Updated: (LUCENE-847) Factor merge policy out of IndexWriter
Date Fri, 24 Aug 2007 16:39:31 GMT


Michael McCandless updated LUCENE-847:

    Attachment: LUCENE-847.take3.patch

OK I started from the original patch and made the changes described

This is still a work in progress, but I think I think the new
stateless approach works very well.

All unit tests pass (one assert had to be changed in

I created a ConcurrentMergePolicyWrapper along with this (I'll post
patch to LUCENE-870).

I've also included the two merge policies from LUCENE-845 (still
defaulting to LogDocMergePolicy).

Here are the changes:

  - Renamed merge -> maybeMerge

  - Changed the API to be "stateless" meaning the merge policy is no
    longer responsible for running the merges itself.  Instead, it
    quickly returns the specification, which describes which merges
    are needed, back to the writer and the writer then runs them.  I
    also changed MergeSpecification to contain a list of OneMerge

  - Removed IndexMerger interface (just use IndexWriter instead)

  - Put isOptimized() logic into LogMergePolicy: on thinking about
    this more (and seeing response to a thread on java-dev), I now
    agree with Steve that this logically belongs in LogMergePolicy
    because each MergePolicy is free to define just what it considers
    "optimized" to mean.  Then I removed the MergePolicyBase.

  - Un-deprecated {get/set}{UseCompoundFile,MergeFactor,MaxMergeDocs}.
    But I did leave the static constants deprecated.

  - IndexWriter keeps track of which segments are involved in running
    merges and throws a MergeException if it's asked to initiate a
    merge that involves a segment that's already being merged.

  - Fixed LogMergePolicy to return all possible merges (exposes

  - Implemented the "merge deletes when commiting the merge" algorithm
    that Ning suggested (this is in commitMerge).

  - Assert that the merge request is in fact contiguous (at start &
    finish of merge) & throw MergeException if not.

  - Fixed a number of sneaky concurrency issues so that CMPW would
    work.  Broke "merge" into mergeInit, mergeMiddle and mergeFinish.
    The first & last are carefully sychronized.

  - I put copyDirFiles in IW and call this in addIndexesNoOptimize
    before committing new segments file: we can't let mergePolicy
    leave the index inconsistent.

  - I reverted the changes to addIndexes(IndexReader[]): I think the
    change here wasn't valid: you can't assume that you can re-create
    any IndexReader instance by loading from its directory; I put the
    original back for this method.

  - the changes to addIndexes I'm not sure are good.

  - Fixed LogMergePolicy to return more than 1 merge

  - Made CMPW

  - Renamed replace -> commitMerge; made it private.

> Factor merge policy out of IndexWriter
> --------------------------------------
>                 Key: LUCENE-847
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Steven Parkes
>            Assignee: Steven Parkes
>         Attachments: concurrentMerge.patch, LUCENE-847.patch.txt, LUCENE-847.patch.txt,
LUCENE-847.take3.patch, LUCENE-847.txt
> If we factor the merge policy out of IndexWriter, we can make it pluggable, making it
possible for apps to choose a custom merge policy and for easier experimenting with merge
policy variants.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message