lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wettin <karl.wet...@gmail.com>
Subject Re: Lucene Partition Size
Date Thu, 08 Apr 2010 18:44:29 GMT

8 apr 2010 kl. 20.05 skrev Ivan Provalov:

> We are using Lucene for searching of 200+ mln documents (periodical  
> publications).  Is there any limitation on the size of the Lucene  
> index (file size, number of docs, etc...)?

The only such limitation in Lucene I'm aware of is Integer.MAX_VALUE  
documents. This might also be true for number of terms.

> We are partitioning the indexes at about 10 mln documents per  
> partition (each partition is on a separate box, some mounted volumes  
> are shared among three-four partitions).  We have index size around  
> 10% of the content size.  The total content is around 4Tb and the  
> index around 400Gb.  This content is planned to be split into 20  
> partitions (10mln docs, 200Gb content size, 20Gb index size).  We  
> are using a memory mapped index directory implementation.  Our  
> testing is done with 600 concurrent users.
>
> We are seeing consistently high response times from the partitions  
> (4-5 seconds).  Is there a number of documents per partition  
> limitation in Lucene for this particular scenario?

I'm not sure if I got this right but it sounds like your index is  
mounted over network? Can you tell us some more details about that?  
What speeds do you see if you put the index on local disc?



	karl

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message