lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Lea <ian....@gmail.com>
Subject Re: Multiple index vs Single Index
Date Thu, 23 Feb 2012 11:25:16 GMT
Well, you certainly can force a merge if you wish,  I guess it's a
balance between an expensive, disk intensive operation that may make
other operations quicker.  Your choice.

I only have one set of multi-million doc indexes whose performance I
care about and they are updated in bulk every night and are optimized
( forceMerge(1)) once a week, on a Sunday.  It's been that way for
years and although the optimize may well now be redundant, I'm happy
to take the weekly hit and maybe gain a bit of speed the rest of the
time.

Your system will no doubt be different and your mileage may vary ...


--
Ian.


On Thu, Feb 23, 2012 at 11:00 AM, Ganesh <emailgane@yahoo.co.in> wrote:
> Thanks.
>
> The reason i have gone for multiple index is, I have more updates for current date and
deletes in the older date. Now i am planning to use Single index. I think i should use forceMergeDelete
to merge the deletes.
>
> Do you optimize your index? How you handle millions of docs in the index.
>
> Regards
> Ganesh
>
> ----- Original Message -----
> From: "Ian Lea" <ian.lea@gmail.com>
> To: <java-user@lucene.apache.org>
> Sent: Thursday, February 23, 2012 3:40 PM
> Subject: Re: Multiple index vs Single Index
>
>
> Millions of docs in a single index is definitely OK.  If it was my
> system I'd willingly trade slightly slower indexing for simplicity and
> ease of use.  If it works and is fast enough, job done.
>
>
> --
> Ian.
>
>
> On Thu, Feb 23, 2012 at 7:31 AM, Ganesh <emailgane@yahoo.co.in> wrote:
>> Hello all,
>>
>> This debate we might have had more frequently in the group. Yet one more time, i
want to clarify.
>>
>> I was using multiple indexes (per week one index) with previous versions of Lucene
(2.4 - 3.0.3). The performance was really good for incremental indexing. I used to optimize
once per day. The only disadvantage is, when data get piled up to 1 year and when i try to
reindex, it is throwing too many open files exception.
>>
>> I have seen couple of comments in the group to use Single Index / per machine and
not use multiple index. The suggesstion is, not to use optimize for large index. This works
fine. I am able to index more than 58 million records. The search performance is really good
but index performance is little slow then in my previous model.
>>
>> Previously, I have choosen multiple indexes model beacause i need to do optimize
and it was failing for the large index. I thought to break in to multiple small index with
same schema. Now optimization is discourged and the name has been changed to forceMerge to
discourge its use.
>>
>> I just want to confirm, Is Single Index would be good for millions of records?
>>
>> Regards
>> Ganesh
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message