lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shai Erera <ser...@gmail.com>
Subject Re: performance merging indexes with addIndexesNoOptimize
Date Fri, 12 Nov 2010 17:22:34 GMT
Ok, so a couple of clarifications:

addIndexes(Directory...) *does not* trigger any merges. It simply registers
the incoming directories in the target index, and returns. You can later
call maybeMerge() or optimize() as you see fit.

Compound files are irrelevant to addIndexes - it just adds the incoming
ones. To later merge them and into a compound file, you'd have to change the
MergePolicy settings to create compound files and then call optimize(). This
is true even if you're not using addIndexes - say you're adding documents to
an index w/ setUseCompoundFiles(false), commits a couple of times, then
setUseCompFiles(true) and call optimize(), they will be converted to
compound files.

Calling w.optimize() will trigger merges and wait for all of them to
complete before the method returns. Therefore calling close() or
close(false) afterwards is the same. Calling optimize(false) followed by a
close(false) (immediately) means that ~0 merges will complete, so that's not
good either.

If you want optimize to finish, call optimize(). If you want to close the
IndexWriter as soon as possible, call close(false). Placing these two method
calls immediately one after the other makes sense only if you call either:
1) optimize(); close(); or
2) optimize(false); close();

In either case, you want to wait for merges. If you don't want that, then
just don't call optimize() can use close(false).

Shai

On Fri, Nov 12, 2010 at 6:56 PM, Marc Sturlese <marc.sturlese@gmail.com>wrote:

>
> Thanks a lot Shai, couple of questions:
>
> >> In Lucene 3x there is a new addIndexes which accepts Directory… that
> >> simply registers the new indexes in the index, without running merges.
> >> That makes addIndexes very fast.
> With the lucene 3.X addIndexes which accepts Directory, if after the merge
> I
> need to optimize the index using compound file, what would happen? Would
> this optimize be slower than if I use the addIndexesNoOptimize?
>
> >> But note that not running merges, or letting them finish, is not
> >> recommended long term. The approach I've mentioned are good if you
> >> want to quickly add new indexes and plan to run index optimization at
> >> a later time.
>
> Actually I always build all the "small indexes" from scratch and have to
> optimize my final index using compound file (the merged one). If i do:
> w.optimze() ;
> w.close (false) ;
> Would I be getting any benefit if the w.close (false) ?
>
> Thanks in advance
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/performance-merging-indexes-with-addIndexesNoOptimize-tp1889378p1890077.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message