lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Desidero <desid...@gmail.com>
Subject Re: Query performance in Lucene 4.x
Date Tue, 01 Oct 2013 21:37:28 GMT
For anyone who was wondering, this was actually resolved in a different
thread today. I misread the information in the
IndexSearcher(IndexReader,ExecutorService) constructor documentation - I
was under the impression that it was submitting a thread for each index
shard (MultiReader wraps 20 shards, so 20 tasks) but it was really
submitting a task for each segment within each shard (20 shards * ~10
segments = ~200 tasks) which is horrible. Since my index changes
infrequently, I'm using forceMerge(1) before sending out updated indexes to
the slave servers. Without any extra tuning (threads, # of shards, etc)
I've gone from ~2900 requests per minute to ~10k requests per minute.

Thanks to Adrien and Mike for the clarification and Benson for bringing up
the question that led to my answer.

I'm still pretty new to Lucene so I have a lot of poking around to do, but
I'm going to try to implement the "virtual segment" concept that Mike
mentioned. It'll be really helpful for those of us who want parallelism
within queries and don't want to forceMerge.


On Fri, Sep 27, 2013 at 9:55 AM, Desidero <desidero@gmail.com> wrote:

> Erick,
>
> Thank you for responding.
>
> I ran tests using both compressed fields and uncompressed fields, and it
> was significantly slower with uncompressed fields. I looked into the lazy
> field loading per your suggestion, but we don't get any values from the
> returned Documents until the result set has been appropriately reduced.
> Since we only store one retrievable field and we always need to get it, it
> doesn't save any time loading it lazily.
>
> I'll try running a test without loading any fields just to see how it
> affects performance and let you know how that goes.
>
> Regards,
> Matt
>
>
> On Fri, Sep 27, 2013 at 8:01 AM, Erick Erickson <erickerickson@gmail.com>wrote:
>
>> Hmmm, since 4.1, fields have been stored compressed by default.
>> I suppose it's possible that this is a result of
>> compressing/uncompressing.
>>
>> What happens if
>> 1> you enable lazy field loading
>> 2> don't load any fields?
>>
>> FWIW,
>> Erick
>>
>> On Thu, Sep 26, 2013 at 10:55 AM, Desidero <desidero@gmail.com> wrote:
>> > A quick update:
>> >
>> > In order to confirm that none of the standard migration changes had a
>> > negative effect on performance, I ported my Lucene 4.x version back to
>> > Lucene 3.6.2 and kept the newer API rather than using the custom
>> > ParallelMultiSearcher and other deprecated methods/classes.
>> >
>> > Performance in 3.6.2 is even faster than before (~2900 requests/min
>> with 4.x
>> > vs ~6200 requests/min with 3.6.2), so none of my code changes should be
>> > causing the difference. It seems to be something Lucene is doing under
>> the
>> > covers.
>> >
>> > Again, if there's any other information if I can provide to help
>> determine
>> > what's going on, please let me know.
>> >
>> > Thanks,
>> > Matt
>> >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message