lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jake Mannix <>
Subject Re: Index splitter
Date Wed, 07 Oct 2009 21:03:56 GMT
As long as you don't have to split up a fully optimized index, or
one with the wrong number of segments for the division you
want to do, that would be useful.  Of course, sometimes you
need to split up the big segments into smaller ones too, but
the only way I've done that in the past is basically:

   1. make two copies of the index
   2. delete the first half of the documents from one copy,
   3. delete the second half of the documents on the other,
   4. purge deleted docs from both,
   5. addIndexesNoOptimize() to add them both to a new index.

Having a generic IndexSplitter which took an index of any number
of segments and redistributed them into either any number of
different sized segments, and optionally, completely different
indexes, now that would indeed be a useful tool.


On Wed, Oct 7, 2009 at 1:50 PM, Jason Rutherglen <
> wrote:

> We have a way to merges indexes together with IW.addIndexes,
> however not the opposite, split up an index with multiple
> segments. I think I can simply manufacture a new segmentinfos in
> a new directory, copy over the segments files from those
> segments, delete the copied segments from the source, and save
> the old segmentinfos less the copied segments. It seems like a
> useful tool? The use case is I've reached a limit on EC2 and the
> index has slowed to a crawl, so rather than reindex 80GB in two
> different places, this approach would be more efficient.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message