lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@apache.org>
Subject Re: Recommended values for mergeFactor, minMergeDocs, maxMergeDocs
Date Mon, 06 Dec 2004 17:34:58 GMT
Chuck Williams wrote:
> I've got about 30k documents and have 3 indexing scenarios:
> 
> 1.       Full indexing and optimize
> 2.       Incremental indexing and optimize
> 3.       Parallel incremental indexing without optimize
> 
> Search performance is critical.  For both cases 1 and 2, I'd like the
> fastest possible indexing time.  For case 3, I'd like minimal pauses and
> no noticeable degradation in search performance.
> 
>  
> 
> Based on reading the code (including the javadocs comments), I'm
> thinking of values along these lines:
> 
> mergeFactor:  1000 during Full indexing, and during optimize (for both
> cases 1 and 2); 10 during incremental indexing (cases 2 and 3)

1000 is too big of a mergeFactor for any practical purpose.

I don't see a point in using different mergeFactors in cases 1 and 2. 
If you're going to optimize before you search, then you want the fastest 
batch indexing mode.  I would use something like 50 for both cases 1 and 2.

For case 3, where unoptimized search performance is very important, I 
would use something smaller than 10.  For Technorati's blog search, 
which incrementally maintains a Lucene index with millions of documents, 
I used a mergeFactor of 2 in order to maximize search performance. 
Indexing performance on a single CPU is still adequate to keep up with 
the rate of change of today's blogosphere.

> minMergeDocs:  1000 during Full indexing, 10 during incremental indexing

I see no reason to lower this when indexing incrementally.  1000 is a 
good value for high performance indexing when RAM is plentiful and 
documents are not too large.

> maxMergeDocs:  Integer.MAX_VALUE during full indexing, 1000 during
> incremental indexing

1000 seems low to me, as it will result in too many segments, slowing 
search.  Here one should select the largest value that can be merged in 
the maximum time delay permitted in your application between a new 
document arriving and it appearing in search results.  So how up-to-date 
must your index be?  If it's okay for it to ocassionally be a few 
minutes out of date, then you can probably safely increase this to at 
least tens or hundreds of thousands, perhaps even millions.  When 
incrementally indexing, the most recently added segments stay cached in 
RAM by the filesystem.  So, on a system with a gigabyte of RAM that's 
dedicated to incremental indexing, you might safely set maxMergeDocs to 
account for a few hundred megabytes of index without encountering slow, 
i/o-bound merges.

> Since mergeFactor is used in both addDocument() and optimize(), I'm
> thinking of using two different values in case 2:  10 during the
> incremental indexing, and then 1000 during the optimize.  Is changing
> the value like this going to cause a problem?

It should not cause problems to use different mergeFactors at different 
times.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message