lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Desidero <desid...@gmail.com>
Subject Re: Query performance in Lucene 4.x
Date Tue, 01 Oct 2013 23:10:57 GMT
Uwe,

I was using a bounded thread pool.

I don't know if the problem was the task overload or something about the
actual efficiency of searching a single segment rather than iterating over
multiple AtomicReaderContexts, but I'd lean toward task overload. I will do
some testing tonight to find out for sure.

Matt
 Hi,

use a bounded thread pool.

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Desidero [mailto:desidero@gmail.com]
> Sent: Tuesday, October 01, 2013 11:37 PM
> To: java-user@lucene.apache.org
> Subject: Re: Query performance in Lucene 4.x
>
> For anyone who was wondering, this was actually resolved in a different
> thread today. I misread the information in the
> IndexSearcher(IndexReader,ExecutorService) constructor documentation - I
> was under the impression that it was submitting a thread for each index
> shard (MultiReader wraps 20 shards, so 20 tasks) but it was really
submitting
> a task for each segment within each shard (20 shards * ~10 segments = ~200
> tasks) which is horrible. Since my index changes infrequently, I'm using
> forceMerge(1) before sending out updated indexes to the slave servers.
> Without any extra tuning (threads, # of shards, etc) I've gone from ~2900
> requests per minute to ~10k requests per minute.
>
> Thanks to Adrien and Mike for the clarification and Benson for bringing up
> the question that led to my answer.
>
> I'm still pretty new to Lucene so I have a lot of poking around to do,
but I'm
> going to try to implement the "virtual segment" concept that Mike
> mentioned. It'll be really helpful for those of us who want parallelism
within
> queries and don't want to forceMerge.
>
>
> On Fri, Sep 27, 2013 at 9:55 AM, Desidero <desidero@gmail.com> wrote:
>
> > Erick,
> >
> > Thank you for responding.
> >
> > I ran tests using both compressed fields and uncompressed fields, and
> > it was significantly slower with uncompressed fields. I looked into
> > the lazy field loading per your suggestion, but we don't get any
> > values from the returned Documents until the result set has been
> appropriately reduced.
> > Since we only store one retrievable field and we always need to get
> > it, it doesn't save any time loading it lazily.
> >
> > I'll try running a test without loading any fields just to see how it
> > affects performance and let you know how that goes.
> >
> > Regards,
> > Matt
> >
> >
> > On Fri, Sep 27, 2013 at 8:01 AM, Erick Erickson
> <erickerickson@gmail.com>wrote:
> >
> >> Hmmm, since 4.1, fields have been stored compressed by default.
> >> I suppose it's possible that this is a result of
> >> compressing/uncompressing.
> >>
> >> What happens if
> >> 1> you enable lazy field loading
> >> 2> don't load any fields?
> >>
> >> FWIW,
> >> Erick
> >>
> >> On Thu, Sep 26, 2013 at 10:55 AM, Desidero <desidero@gmail.com>
> wrote:
> >> > A quick update:
> >> >
> >> > In order to confirm that none of the standard migration changes had
> >> > a negative effect on performance, I ported my Lucene 4.x version
> >> > back to Lucene 3.6.2 and kept the newer API rather than using the
> >> > custom ParallelMultiSearcher and other deprecated methods/classes.
> >> >
> >> > Performance in 3.6.2 is even faster than before (~2900 requests/min
> >> with 4.x
> >> > vs ~6200 requests/min with 3.6.2), so none of my code changes
> >> > should be causing the difference. It seems to be something Lucene
> >> > is doing under
> >> the
> >> > covers.
> >> >
> >> > Again, if there's any other information if I can provide to help
> >> determine
> >> > what's going on, please let me know.
> >> >
> >> > Thanks,
> >> > Matt
> >> >
> >> >
> >> >
> >> > -------------------------------------------------------------------
> >> > -- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message