lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dror Matalon <d...@zapatec.com>
Subject Re: index size question
Date Wed, 14 Jan 2004 17:27:14 GMT
It's in the jguru FAQ
http://www.jguru.com/faq/view.jsp?EID=538304

By the way, what platforms don't support files greater than 2GB in this
day and age?

Answer
This question is often brought up because of the 2GB file size limit of
some 32-bit operating systems.

This is a slightly modified answer from Doug Cutting:

The easiest thing is to set IndexWriter.maxMergeDocs.
If, for instance, you hit the 2GB limit at 8M documents set maxMergeDocs
to 7M. That will keep Lucene from trying to merge an index that won't
fit in your filesystem. It will actually effectively round this down to
the next lower power of Index.mergeFactor.
So with the default mergeFactor set to 10 and maxMergeDocs set to 7M
Lucene will generate a series of 1M document indexes, since merging 10
of these would exceed the maximum.

A slightly more complex solution:
You could further minimize the number of segments if, when you've added
7M documents, optimize the index and start a new index. Then use
MultiSearcher to search the indexes.

An even more complex and optimal solution:
Write a version of FSDirectory that, when a file exceeds 2GB, creates a
subdirectory and represents the file as a series of files.

Is this item helpful?  yes  no     Previous votes   Yes: 1  No: 0


On Wed, Jan 14, 2004 at 10:50:48AM -0500, Chong, Herb wrote:
> this should probably in the FAQ. what happens when i index tens of gigabytes of documents
on a platform that doesn't support files larger than 2GB. does Lucene automatically stop merging
index files intelligently so that its files don't exceed 2GB in size, or must i manage the
incoming documents such that no index file exceeds 2GB?
> 
> Herb....
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> 

-- 
Dror Matalon
Zapatec Inc 
1700 MLK Way
Berkeley, CA 94709
http://www.fastbuzz.com
http://www.zapatec.com

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message