lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Lea <ian....@gmail.com>
Subject Re: Merge policy, optimization for small frequently changing indexes.
Date Tue, 05 Oct 2010 10:29:07 GMT
Have you considered having fewer indexes, each storing data for
multiple users?  Obviously with some indexed field that you can use
for restricting searches to data for that user.

I believe that is more common practice for this sort of scenario and
is known to work well.  You seem to be adding possibly unnecessary
complexity and overhead for all the tiny indexes.


--
Ian.


On Tue, Oct 5, 2010 at 11:18 AM, Naveen Kumar <id.navs@gmail.com> wrote:
> Thank you, Ian
> I have a large number of dynamically changing Index, so calling
> expungeDeletes() and optimize() is very costly.
> At this point I am opting to just set a optimum merge factor and skip
> optimize()
>
> On Tue, Oct 5, 2010 at 2:54 PM, Ian Lea <ian.lea@gmail.com> wrote:
>
>> Deleted docs will be removed by lucene at some point - there is no
>> need to run optimize.
>> Read the javadocs for IndexWriter for details.  See also
>> expungeDeletes().  That may be just what you need.
>>
>>
>> --
>> Ian.
>>
>>
>> On Tue, Oct 5, 2010 at 7:48 AM, Naveen Kumar <id.navs@gmail.com> wrote:
>> > Hi
>> > I have one more question, does Lucene purge the deleted documents before
>> > merging the segments, or purging of deleted  documents done only when
>> > optimized?
>> >
>> > On Thu, Sep 30, 2010 at 4:59 PM, Naveen Kumar <id.navs@gmail.com> wrote:
>> >
>> >> Hi
>> >> I have a Very large number (say 3 million) of frequently changing Small
>> >> indexes. 90% of these indexes contain about 50 documents, while a few
>> 2-3%
>> >> indexes have about 100,000 documents each (these being the more
>> frequently
>> >> used indexes).
>> >> Each index belongs to a signed in user, thus can have unpredictable
>> index
>> >> changes, in short periods of time, or can be stagnant for a long time.
>> >> What kind of indexing policy (mergefactor, maxmergedocs) would be
>> optimal
>> >> for this kind of index. Is optimizing for this kind of index needed? or
>> >> wise?, if yes, what would be a good way to optimize (please note number
>> of
>> >> indexes are very large).
>> >>
>> >> Any suggestions would be very helpful.
>> >>
>> >> --
>> >> Thanks
>> >> Naveen Kumar
>> >>
>> >>
>> >>
>> >>
>> >
>> >
>> > --
>> > Thanks
>> > Naveen Kumar
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
> Naveen Kumar
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message