lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Re: best mergeFactor for merging 100 Indexes
Date Fri, 25 Jun 2004 16:30:44 GMT
If this is an option, use compund index format
(writer.setUseCompound(true)).  ulimit -a in some UNIX shells will tell
you the max number of open files allowed.  If you can, increase that
number as high as you can.  Of course, how high you can go also depends
on your RAM.  Finally, don't forget there is a minMergeDocs parameter
you can use to tune things, too.

Otis


--- Harald Kirsch <kirsch@ebi.ac.uk> wrote:
> Hi,
> 
> after an hour of indexing on a cluster I got 100 Indexes, ca. 25MB
> each, 2 indexed fields. I intend now to run code roughly like
> 
>    IndexWriter writer = new IndexWriter(destDir, ...);
>    writer.addIndexes(my100IndexDirs);
>    writer.close()
> 
> When I did this a year ago, I know I had tough problems getting
> around
> memory limitations and open file limits at the same time. In the end
> it worked with writer.mergeFactor==4000, but I think it was a
> specially tweaked kernel on Linux which I don't have anymore.
> 
> Since I don't really understand yet how open files, segments, memory
> use, indexing time and mergeFacter interact, I would appreciate a
> good
> gues how to combine these indexes.
> 
> Which mergeFactor to use?
> Use a different strategy then the 3 lines shown above?
> 
>   Thanks,
>   Harald.
> 
> -- 
>
------------------------------------------------------------------------
> Harald Kirsch | kirsch@ebi.ac.uk | +44 (0) 1223/49-2593
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message