lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Wang (JIRA)" <>
Subject [jira] Commented: (LUCENE-1634) LogMergePolicy should use the number of deleted docs when deciding which segments to merge
Date Wed, 13 May 2009 16:17:45 GMT


John Wang commented on LUCENE-1634:

Comment on implementing a custom merge policy:
As the API current stands, I think the behavior is to assume a subclass of LogMergePolicy.
And one cannot subclass LogMergePolicy without injecting the class into the org.apache.lucene.index
package, (because the api signature: size(org.apache.lucene.index.SegmentInfo info), SegmentInfo
is not an exposed API.

> LogMergePolicy should use the number of deleted docs when deciding which segments to
> ------------------------------------------------------------------------------------------
>                 Key: LUCENE-1634
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Yasuhiro Matsuda
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 2.9
>         Attachments: LUCENE-1634.patch
> I found that IndexWriter.optimize(int) method does not pick up large segments with a
lot of deletes even when most of the docs are deleted. And the existence of such segments
affected the query performance significantly.
> I created an index with 1 million docs, then went over all docs and updated a few thousand
at a time.  I ran optimize(20) occasionally. What saw were large segments with most of docs
deleted. Although these segments did not have valid docs they remained in the directory for
a very long time until more segments with comparable or bigger sizes were created.
> This is because LogMergePolicy.findMergeForOptimize uses the size of segments but does
not take the number of deleted documents into consideration when it decides which segments
to merge. So, a simple fix is to use the delete count to calibrate the segment size. I can
create a patch for this.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message