lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charles Lloyd <>
Subject Re: Large Indexes
Date Mon, 14 Nov 2005 05:12:01 GMT
On Nov 13, 2005, at 8:19 PM, Friedland, Zachary (EDS - Strategy) wrote:
> What is the largest lucene index that has been built?  We're looking to
> build a sort of data warehouse that will hold transaction log files as
> long as possible.  This index would grow at the rate of 10 million
> documents per month indefinitely.  Is there a limit where lucene will
> fail?  What should we expect to happen as this index grows?

Our index has about 3 million documents and is about 11 Gig in size.  
It has become unwieldy to manage at that size so I have been working to 
reduce the size of our Stored data and it looks like we're going to get 
the index size down below 2 Gig.  The overall performance directly 
seems to correlate the to the size of the index.  Try to keep your 
Stored fields minimal as it would appear that the number of documents 
is far less an issue than the physical size of the index file.  At some 
point, we will probably start to split our index if it gets above 5 Gig 
again.  However, my initial tests with the MultiReader have been 
disappointing.  Hopefully, a faster MultiReader will be available by 
the time we cross that bridge.


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message