lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julien Nioche" <Julien.Nio...@lingway.com>
Subject Re: Too Many Open Files
Date Thu, 09 Oct 2003 08:51:41 GMT
Hello Wilton,

MergeFactor indicates when Segments must be merged. With a value of 10, the
segments of your index will be merged into a single one when they reach this
number. That's why MergeFactor has a direct impact on the number of files in
your index.

However this has consequences on the performance if you are using
FSDirectory. When you're adding a new Document, it is stored in RAM and then
merged according to the MergeFactor specified. If this MergeFactor is low,
this will produce Segments in FSDirectory more often and your indexing takes
more time.

If you need a faster indexation there are 3 possibilities  :
- indexing in several RAMDirectories and then merging the indexes in
FSDirectory (see the excellent article by Otis "Advanced Text Indexing" )
- see the FileHandle solution proposed by Dmitry Serebrennikov in September
(which gives the possibility to use a bigger MergeFactor but aggregates the
Segments into a single file)
- and at last use the patch I submitted 9/20/2003 that modifies IndexWriter
by adding a new parameter minMergeDocs which specifies the number of
Documents merged in RAM independently of the number of Segments in
FSDirectory. You still have a fast indexation by using memory and control
the number of files on the disk.

Hope it helps

Julien  Nioche
http://www.lingway.com


----- Original Message -----
From: "Wilton, Reece" <Reece.Wilton@dig.com>
To: "Lucene Users List" <lucene-user@jakarta.apache.org>
Sent: Thursday, October 09, 2003 12:21 AM
Subject: RE: Too Many Open Files


Ok, I lowered my MergeFactor down to 10 and am re-indexing (takes
several hours).  Will lowering the MergeFactor reduce the total number
of files in the index directory?
-Reece

-----Original Message-----
From: Doug Cutting [mailto:cutting@lucene.com]
Sent: Tuesday, October 07, 2003 8:20 PM
To: Lucene Users List
Subject: Re: Too Many Open Files

Wilton, Reece wrote:
> The index directory that Lucene created has 2,322 files in it.  When I
> try to open it I get the dreaded "Too Many Open Files" problem:
>     java.io.FileNotFoundException: C:\Index\_1lvq.f107 (Too many open
> files)
>
> The index has about 50,000 docs in it.  It was created with a merge
> factor of 5,000.  Is there a way that I can reduce the number of files
> or increase the number of files that windows can open?

5000 is way too large for the merge factor.  Please read the FAQ and
other messages on this list for guidelines.  I've personally never found

use for a merge factor larger than 50.

Doug


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message