lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-994) Change defaults in IndexWriter to maximize "out of the box" performance
Date Mon, 01 Oct 2007 07:58:52 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12531433
] 

Michael McCandless commented on LUCENE-994:
-------------------------------------------


> While trying Solr with the latest Lucene, I ran into this back-incompatibility:
> Caused by: java.lang.IllegalArgumentException: this method can only be called when the
merge policy is LogDocMergePolicy
> at org.apache.lucene.index.IndexWriter.getLogDocMergePolicy(IndexWriter.java:316)
> at org.apache.lucene.index.IndexWriter.setMaxMergeDocs(IndexWriter.java:768)
>
> It's not an issue at all for Solr - we'll fix things up when we
> officially upgrade Lucene versions, but it does seem like it might
> affect a number of apps that try and just drop in a new lucene
> jar. Thoughts?

Hmm, good catch.

This should only happen when "setMaxMergeDocs" is called (this is the
only method that requires a LogDocMergePolicy).  I think we have
various options:

  1. Leave things as is and put up-front comment in the release saying
     you could either switch to LogDocMergePolicy, or, use
     "setMaxMergeMB" on the default LogByteSizeMergePolicy, instead.
     Also put details in the javadocs for this method explaining these
     options.

  2. Switch back to LogDocMergePolicy by default "out of the box".

  3. If setMaxMergeDocs() is called, switch back to LogDocMergePolicy
     "on-demand".

  4. Modify LogByteSizeMergePolicy to in fact accept both
     "maxMergeDocs" or "maxMergeMB", allowing either one or both just
     like "flush by RAM" and/or "flush by doc count" is being done in
     LUCENE-1007.

I think I like option 4 the best.  3 seems to magical (violates
"principle of least surprise").  2 I think is bad because it's best to
match the merge policy with how we are flushing (by RAM by default).
1 is clearly disruptive to people who want to drop Lucene JAR in and
test.

I'll open a new issue.  Thanks Yonik!

> Change defaults in IndexWriter to maximize "out of the box" performance
> -----------------------------------------------------------------------
>
>                 Key: LUCENE-994
>                 URL: https://issues.apache.org/jira/browse/LUCENE-994
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.3
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 2.3
>
>         Attachments: LUCENE-994.patch, writerinfo.zip
>
>
> This is follow-through from LUCENE-845, LUCENE-847 and LUCENE-870;
> I'll commit this once those three are committed.
> Out of the box performance of IndexWriter is maximized when flushing
> by RAM instead of a fixed document count (the default today) because
> documents can vary greatly in size.
> Likewise, merging performance should be faster when merging by net
> segment size since, to minimize the net IO cost of merging segments
> over time, you want to merge segments of equal byte size.
> Finally, ConcurrentMergeScheduler improves indexing speed
> substantially (25% in a simple initial test in LUCENE-870) because it
> runs the merges in the backround and doesn't block
> add/update/deleteDocument calls.  Most machines have concurrency
> between CPU and IO and so it makes sense to default to this
> MergeScheduler.
> Note that these changes will break users of ParallelReader because the
> parallel indices will no longer have matching docIDs.  Such users need
> to switch IndexWriter back to flushing by doc count, and switch the
> MergePolicy back to LogDocMergePolicy.  It's likely also necessary to
> switch the MergeScheduler back to SerialMergeScheduler to ensure
> deterministic docID assignment.
> I think the combination of these three default changes, plus other
> performance improvements for indexing (LUCENE-966, LUCENE-843,
> LUCENE-963, LUCENE-969, LUCENE-871, etc.) should make for some sizable
> performance gains Lucene 2.3!

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message