lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Willnauer <simon.willna...@googlemail.com>
Subject Re: Large indexes
Date Fri, 08 Jul 2011 15:57:33 GMT
On Fri, Jul 8, 2011 at 4:50 PM, Ian Lea <ian.lea@gmail.com> wrote:
> There are lots of general tips at
> http://wiki.apache.org/lucene-java/ImproveSearchingSpeed.
>
> What version of lucene?  Recent releases should be faster. Have you
> tried with one big index? If everything is running on the same server
> that may well be faster.
>
> Even on single indexes, response of a few seconds sounds high. Are the
> searches complex?  Are you sorting?  Reading lots of hits for each
> search?
>
>
> --
> Ian.
>
> On Fri, Jul 8, 2011 at 3:39 PM, Chris Bamford
> <chris.bamford@talktalk.net> wrote:
>>
>>  Hi
>>
>>
>> I was wondering how to improve search performance over a set of indexes like this:
>>
>>  27G    K1-1/index
>>  19G    K1-2/index
>>  24G    K1-3/index
>>  15G    K1-4/index
>>  19G    K1-5/index
>>  31G    K2-1/index
>>  16G    K2-2/index
>>  8.1G   K2-3/index
>>  12G    K2-4/index
>>  15G    K2-5/index
>>
>> In total it is ~192Gb.  All have the same schema, with data partitioned by my application
into separate
>> indexes (rather than just 1 big one).
>> I have tried searching with both a MultiSearcher and a ParallelMultiSearcher (with
lots of memory), but it
>> is way too slow (sometimes hangs, otherwise it takes tens of minutes).  Do you have
any tips for improving speed?
>> I have reduced the stored data as much as possible (without using compression) and
use a FieldSelector when querying
>> but neither seem to help..
>>
>> Is this the right approach?  If I query them one by one I response time is reasonable
(a few seconds each), but I was
>> kind of hoping to treat them as one large index if possible.

a few seconds? what are you doing that a search takes a few seconds.

simon
>>
>> Any thoughts welcomed.
>>
>> Thanks,
>>
>>  - Chris
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message