lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] Updated: (LUCENE-2773) Don't create compound file for large segments by default
Date Wed, 24 Nov 2010 11:08:13 GMT

     [ https://issues.apache.org/jira/browse/LUCENE-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Michael McCandless updated LUCENE-2773:
---------------------------------------

    Fix Version/s: 3.0.3
                   2.9.4

I'll commit this soon to trunk, but...

I think we should also back-port it to 2.9.x/3.0.x.

On the one hand, it's a sizable change to IndexWriter's defaults, in that suddenly, if you
use CFS, you'll see your large segments no longer being converted to CFS, and if you have
a large index you'll see your large segments no longer getting merged away due to the change
to maxMergeMB.  Though, these decisions have always been "under the hood", so the change the
app sees would be eg on listing the directory, and not really on any "external" factors.

But, on the other hand, if we don't back port, then suddenly large merges require substantially
more transient peak disk space than before, which is a very external change.

So, it's a lesser-of-evils situation, and I think the lesser evil is to change the defaults.

> Don't create compound file for large segments by default
> --------------------------------------------------------
>
>                 Key: LUCENE-2773
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2773
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 2.9.4, 3.0.3, 3.1, 4.0
>
>         Attachments: LUCENE-2773.patch
>
>
> Spinoff from LUCENE-2762.
> CFS is useful for keeping the open file count down.  But, it costs
> some added time during indexing to build, and also ties up temporary
> disk space, causing eg a large spike on the final merge of an
> optimize.
> Since MergePolicy dictates which segments should be CFS, we can
> change it to only build CFS for "smallish" merges.
> I think we should also set a maxMergeMB by default so that very large
> merges aren't done.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message