lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-1634) LogMergePolicy should use the number of deleted docs when deciding which segments to merge
Date Wed, 13 May 2009 16:21:45 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708991#action_12708991
] 

Michael McCandless commented on LUCENE-1634:
--------------------------------------------

bq. This is actually referring to the optimize(int) call

Ahh, woops -- sorry, I missed that the first time around.  But I don't think the patch addresses
how optimize(int) selects its merges?

> LogMergePolicy should use the number of deleted docs when deciding which segments to
merge
> ------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-1634
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1634
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Yasuhiro Matsuda
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 2.9
>
>         Attachments: LUCENE-1634.patch
>
>
> I found that IndexWriter.optimize(int) method does not pick up large segments with a
lot of deletes even when most of the docs are deleted. And the existence of such segments
affected the query performance significantly.
> I created an index with 1 million docs, then went over all docs and updated a few thousand
at a time.  I ran optimize(20) occasionally. What saw were large segments with most of docs
deleted. Although these segments did not have valid docs they remained in the directory for
a very long time until more segments with comparable or bigger sizes were created.
> This is because LogMergePolicy.findMergeForOptimize uses the size of segments but does
not take the number of deleted documents into consideration when it decides which segments
to merge. So, a simple fix is to use the delete count to calibrate the segment size. I can
create a patch for this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message