Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of erickerickson@gmail.com
 designates 209.85.216.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=E6Iq2lZfJCEt9897PAGqfmMfUs4rkPCs4/ffPIPI7TqOKKzuhOp3ws2K51i9l+GGQq
         7XbwqoohfK96OXQKcVGJwiHrtmxruEL3Q8UQ32lV56imC2h3sXBksq8LCARJiwPTGIx/
         YmIqJ1cHcS0Xb+L9Bw9l47jURigltm4iEMveQ=
MIME-Version: 1.0
In-Reply-To: <4DE88196.501@gmail.com>
References: <4DE6A278.4030002@gmail.com>
	<BANLkTi=8AvxMeF86mMBhkA1j5rxHi-VgyQ@mail.gmail.com>
	<4DE6B10F.7090207@gmail.com>
	<BANLkTimsBoV68F5FdDqmvp9-CMRF3w-AVQ@mail.gmail.com>
	<4DE78FE3.2070106@gmail.com>
	<BANLkTikf7xtc-2fwYJwMqB6D0agEKD9vTw@mail.gmail.com>
	<4DE794B5.2010506@gmail.com>
	<4DE88196.501@gmail.com>
Date: Fri, 3 Jun 2011 16:59:44 -0400
Message-ID: <BANLkTi=tAiKo5TS6_TeFFcFZsyVtZK-sLw@mail.gmail.com>
Subject: Re: multiple small indexes or one big index?
From: Erick Erickson <erickerickson@gmail.com>
To: java-user@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1

OK, if they're all in a single index, you might also try using Lucene
sorting. Be aware
that the first sort on a field takes extra time to warm the caches...

But note that sorting is for single-valued, un-tokenized fields..

Best
Erick

On Fri, Jun 3, 2011 at 2:39 AM, Alexander Rosemann
<alexander.rosemann@gmail.com> wrote:
> Alright. With all the changes you suggested I am down from 9s to <1s. Again,
> many thanks to both of you Erick and Shai!
>
> Regards,
> Alex
>
> On 02.06.2011 15:48, Alexander Rosemann wrote:
>>
>> No worries, I'll keep that in mind now.
>> In addition I am going to switch to another collector as well. ATM I
>> collect the results and then sort them using the std. Collections.sort
>> approach... I have to look what Lucene offers and switch to something
>> else.
>>
>> Thanks,
>> Alex
>>
>> On 02.06.2011 15:36, Erick Erickson wrote:
>>>
>>> Sounds good, just be sure to keep your (now single) searcher open! Also,
>>> be sure to measure queries after a while. The first few queries will
>>> fill up
>>> caches etc, so the time should improve after the first few.
>>>
>>> Best
>>> Erick
>>>
>>> On Thu, Jun 2, 2011 at 9:28 AM, Alexander Rosemann
>>> <alexander.rosemann@gmail.com> wrote:
>>>>
>>>> Hi Erick, caching the IndexSearchers didn't took too much effort and
>>>> decreased searching already by 30%!
>>>>
>>>> I am busy changing the code to use a single index as you suggested atm.
>>>> Still a few things left to be done but once I have it working I let
>>>> you know
>>>> how much faster it is for me.
>>>>
>>>> Thanks,
>>>> Alex
>>>>
>>>> On 02.06.2011 13:04, Erick Erickson wrote:
>>>>>
>>>>> At this size, really consider going to a single index. The lack of
>>>>> administrative headaches alone is probably well worth the effort....
>>>>>
>>>>> I almost guarantee that the time you spend re-writing things to keep
>>>>> the searchers open (and finding the bugs!) will be far more than just
>>>>> putting all the data in a single index.
>>>>>
>>>>> But that might just be my preferences showing....
>>>>>
>>>>> Best
>>>>> Erick
>>>>>
>>>>> On Wed, Jun 1, 2011 at 5:37 PM, Alexander Rosemann
>>>>> <alexander.rosemann@gmail.com> wrote:
>>>>>>
>>>>>> Many thanks for the tips, Erick! I do close each searcher after a
>>>>>> search...
>>>>>> I will change that first thing tmrw. and let you know how that went.
>>>>>> Multi-threaded searching will be next and if that hasn't helped, I
>>>>>> will
>>>>>> switch to one big index.
>>>>>> All indexes together are rather small, ~200MB and 50.000 documents.
>>>>>>
>>>>>> -Alex
>>>>>>
>>>>>> On 01.06.2011 23:26, Erick Erickson wrote:
>>>>>>>
>>>>>>> I'd start by putting them all in one index. There's no penalty
>>>>>>> in Lucene for having empty fields in a document, unlike an
>>>>>>> RDBMS.
>>>>>>>
>>>>>>> Alternately, if you're opening then closing searchers each
>>>>>>> time, that's very expensive. Could you open the searchers
>>>>>>> once and keep them open (all 90 of them)? That alone might
>>>>>>> do the trick and be less of a change to your program. You
>>>>>>> could also fire multiple threads at the searches, but check if
>>>>>>> you're CPU bound first (if you are, multiple threads won't
>>>>>>> help much/at all).
>>>>>>>
>>>>>>> You haven't said how big these indexes are nor how many
>>>>>>> documents you're talking about here, so this advice is suspect.
>>>>>>>
>>>>>>> Do look at putting it all in one index though, let us know if you
>>>>>>> have some data indicating how big stuff is/would be.
>>>>>>>
>>>>>>> Best
>>>>>>> Erick
>>>>>>>
>>>>>>> On Wed, Jun 1, 2011 at 4:35 PM, Alexander Rosemann
>>>>>>> <alexander.rosemann@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hi all, I was wondering whether you could give me some advice on
>>>>>>>> how to
>>>>>>>> improve my search performance.
>>>>>>>>
>>>>>>>> I have 90 lucene indexes, each having different fields (~5 per
>>>>>>>> Document).
>>>>>>>> When I search, I always have to go through all indexes to build my
>>>>>>>> result
>>>>>>>> set. Searching one index takes approx. 100ms, thus searching all
>>>>>>>> indexes
>>>>>>>> takes 9s in total.
>>>>>>>>
>>>>>>>> How can I reduce the time it needs to search?
>>>>>>>>
>>>>>>>> I decided to create this many indexes because putting all data in
>>>>>>>> one
>>>>>>>> index
>>>>>>>> would mean that a document would have ~400 fields, with most of them
>>>>>>>> left
>>>>>>>> empty. Is that ok? Would a single index be faster compared to
>>>>>>>> multiple
>>>>>>>> small
>>>>>>>> ones?
>>>>>>>>
>>>>>>>> Any pointers are much appreciated.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Alex
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>
>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org