lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Keegan <peterlkee...@gmail.com>
Subject Re: [jira] Resolved: (LUCENE-1316) Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer
Date Fri, 06 Feb 2009 23:27:23 GMT
I just ran into the same bottleneck with ValueSourceScorer.
Here's a stack trace:

Name: QueryThread group 1,#4
State: BLOCKED on org.apache.lucene.index.SegmentReader@49d7fb83 owned by:
QueryThread group 1,#8
Total blocked: 1,535,881  Total waited: 1,080

Stack trace:
org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:663)
org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:221)
org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.next(ValueSourceQuery.java:138)
org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.skipTo(ValueSourceQuery.java:159)
org.apache.lucene.search.function.CustomScoreQuery$CustomScorer.skipTo(CustomScoreQuery.java:399)
org.apache.lucene.search.ConjunctionScorer.doNext(ConjunctionScorer.java:59)
org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:51)
org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
org.apache.lucene.search.Searcher.search(Searcher.java:118)

Peter


On Sun, Jan 25, 2009 at 9:41 AM, Michael McCandless (JIRA)
<jira@apache.org>wrote:

>
>     [
> https://issues.apache.org/jira/browse/LUCENE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel]
>
> Michael McCandless resolved LUCENE-1316.
> ----------------------------------------
>
>    Resolution: Fixed
>
> Committed revision 737513.  Thanks everyone!
>
> > Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer
> > ------------------------------------------------------------------------
> >
> >                 Key: LUCENE-1316
> >                 URL: https://issues.apache.org/jira/browse/LUCENE-1316
> >             Project: Lucene - Java
> >          Issue Type: Bug
> >          Components: Query/Scoring
> >    Affects Versions: 2.3
> >         Environment: All
> >            Reporter: Todd Feak
> >            Assignee: Michael McCandless
> >            Priority: Minor
> >             Fix For: 2.9
> >
> >         Attachments: LUCENE-1316.patch, LUCENE-1316.patch,
> LUCENE-1316.patch, LUCENE_1316.patch, LUCENE_1316.patch, LUCENE_1316.patch,
> MatchAllDocsQuery.java
> >
> >   Original Estimate: 1h
> >  Remaining Estimate: 1h
> >
> > The isDeleted() method on IndexReader has been mentioned a number of
> times as a potential synchronization bottleneck. However, the reason this
>  bottleneck occurs is actually at a higher level that wasn't focused on (at
> least in the threads I read).
> > In every case I saw where a stack trace was provided to show the
> lock/block, higher in the stack you see the MatchAllScorer.next() method. In
> Solr paricularly, this scorer is used for "NOT" queries. We saw incredibly
> poor performance (order of magnitude) on our load tests for NOT queries, due
> to this bottleneck. The problem is that every single document is run through
> this isDeleted() method, which is synchronized. Having an optimized index
> exacerbates this issues, as there is only a single SegmentReader to
> synchronize on, causing a major thread pileup waiting for the lock.
> > By simply having the MatchAllScorer see if there have been any deletions
> in the reader, much of this can be avoided. Especially in a read-only
> environment for production where you have slaves doing all the high load
> searching.
> > I modified line 67 in the MatchAllDocsQuery
> > FROM:
> >   if (!reader.isDeleted(id)) {
> > TO:
> >   if (!reader.hasDeletions() || !reader.isDeleted(id)) {
> > In our micro load test for NOT queries only, this was a major performance
> improvement.  We also got the same query results. I don't believe this will
> improve the situation for indexes that have deletions.
> > Please consider making this adjustment for a future bug fix release.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

Mime
View raw message