lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Bamford <chris.bamf...@talktalk.net>
Subject Large indexes
Date Fri, 08 Jul 2011 14:39:15 GMT

 Hi 

 
I was wondering how to improve search performance over a set of indexes like this:

 27G    K1-1/index
 19G    K1-2/index
 24G    K1-3/index
 15G    K1-4/index
 19G    K1-5/index
 31G    K2-1/index
 16G    K2-2/index
 8.1G   K2-3/index
 12G    K2-4/index
 15G    K2-5/index

In total it is ~192Gb.  All have the same schema, with data partitioned by my application
into separate 
indexes (rather than just 1 big one).
I have tried searching with both a MultiSearcher and a ParallelMultiSearcher (with lots of
memory), but it 
is way too slow (sometimes hangs, otherwise it takes tens of minutes).  Do you have any tips
for improving speed?
I have reduced the stored data as much as possible (without using compression) and use a FieldSelector
when querying
but neither seem to help..

Is this the right approach?  If I query them one by one I response time is reasonable (a few
seconds each), but I was 
kind of hoping to treat them as one large index if possible.

Any thoughts welcomed.

Thanks,

 - Chris


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message