lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Large indexes
Date Fri, 08 Jul 2011 16:24:01 GMT
Simply breaking up your index into separate pieces on the same machine
buys you nothing, in fact it costs you considerably. Have you put
a profiler on the system to see what's happening? I expect you're swapping
all over the place and are memory-constrained.

Have you considered sharding your index across multiple machines? That
is often an approach to use assuming that logically this is a single index...

Best
Erick

On Fri, Jul 8, 2011 at 10:39 AM, Chris Bamford
<chris.bamford@talktalk.net> wrote:
>
>  Hi
>
>
> I was wondering how to improve search performance over a set of indexes like this:
>
>  27G    K1-1/index
>  19G    K1-2/index
>  24G    K1-3/index
>  15G    K1-4/index
>  19G    K1-5/index
>  31G    K2-1/index
>  16G    K2-2/index
>  8.1G   K2-3/index
>  12G    K2-4/index
>  15G    K2-5/index
>
> In total it is ~192Gb.  All have the same schema, with data partitioned by my application
into separate
> indexes (rather than just 1 big one).
> I have tried searching with both a MultiSearcher and a ParallelMultiSearcher (with lots
of memory), but it
> is way too slow (sometimes hangs, otherwise it takes tens of minutes).  Do you have
any tips for improving speed?
> I have reduced the stored data as much as possible (without using compression) and use
a FieldSelector when querying
> but neither seem to help..
>
> Is this the right approach?  If I query them one by one I response time is reasonable
(a few seconds each), but I was
> kind of hoping to treat them as one large index if possible.
>
> Any thoughts welcomed.
>
> Thanks,
>
>  - Chris
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message