Mailing-List: contact lucene-user-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Lucene Users List" <lucene-user@jakarta.apache.org>
Message-ID: <20040625163044.93039.qmail@web12704.mail.yahoo.com>
Date: Fri, 25 Jun 2004 09:30:44 -0700 (PDT)
From: Otis Gospodnetic <otis_gospodnetic@yahoo.com>
Subject: Re: best mergeFactor for merging 100 Indexes
To: Lucene Users List <lucene-user@jakarta.apache.org>
In-Reply-To: <20040625160043.H10707@neptune.ebi.ac.uk>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

If this is an option, use compund index format
(writer.setUseCompound(true)).  ulimit -a in some UNIX shells will tell
you the max number of open files allowed.  If you can, increase that
number as high as you can.  Of course, how high you can go also depends
on your RAM.  Finally, don't forget there is a minMergeDocs parameter
you can use to tune things, too.

Otis


--- Harald Kirsch <kirsch@ebi.ac.uk> wrote:
> Hi,
> 
> after an hour of indexing on a cluster I got 100 Indexes, ca. 25MB
> each, 2 indexed fields. I intend now to run code roughly like
> 
>    IndexWriter writer = new IndexWriter(destDir, ...);
>    writer.addIndexes(my100IndexDirs);
>    writer.close()
> 
> When I did this a year ago, I know I had tough problems getting
> around
> memory limitations and open file limits at the same time. In the end
> it worked with writer.mergeFactor==4000, but I think it was a
> specially tweaked kernel on Linux which I don't have anymore.
> 
> Since I don't really understand yet how open files, segments, memory
> use, indexing time and mergeFacter interact, I would appreciate a
> good
> gues how to combine these indexes.
> 
> Which mergeFactor to use?
> Use a different strategy then the 3 lines shown above?
> 
>   Thanks,
>   Harald.
> 
> -- 
>
------------------------------------------------------------------------
> Harald Kirsch | kirsch@ebi.ac.uk | +44 (0) 1223/49-2593
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org