lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: [jira] Resolved: (LUCENE-1316) Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer
Date Fri, 06 Feb 2009 23:44:04 GMT

OK I created https://issues.apache.org/jira/browse/LUCENE-1538.

Mike

Peter Keegan wrote:

> I just ran into the same bottleneck with ValueSourceScorer.
> Here's a stack trace:
>
> Name: QueryThread group 1,#4
> State: BLOCKED on org.apache.lucene.index.SegmentReader@49d7fb83  
> owned by: QueryThread group 1,#8
> Total blocked: 1,535,881  Total waited: 1,080
>
> Stack trace:
> org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java: 
> 663)
> org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:221)
> org.apache.lucene.search.function.ValueSourceQuery 
> $ValueSourceScorer.next(ValueSourceQuery.java:138)
> org.apache.lucene.search.function.ValueSourceQuery 
> $ValueSourceScorer.skipTo(ValueSourceQuery.java:159)
> org.apache.lucene.search.function.CustomScoreQuery 
> $CustomScorer.skipTo(CustomScoreQuery.java:399)
> org 
> .apache 
> .lucene.search.ConjunctionScorer.doNext(ConjunctionScorer.java:59)
> org 
> .apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java: 
> 51)
> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
> org.apache.lucene.search.Searcher.search(Searcher.java:118)
>
> Peter
>
>
> On Sun, Jan 25, 2009 at 9:41 AM, Michael McCandless (JIRA) <jira@apache.org 
> > wrote:
>
>     [ https://issues.apache.org/jira/browse/LUCENE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

>  ]
>
> Michael McCandless resolved LUCENE-1316.
> ----------------------------------------
>
>    Resolution: Fixed
>
> Committed revision 737513.  Thanks everyone!
>
> > Avoidable synchronization bottleneck in MatchAlldocsQuery 
> $MatchAllScorer
> >  
> ------------------------------------------------------------------------
> >
> >                 Key: LUCENE-1316
> >                 URL: https://issues.apache.org/jira/browse/LUCENE-1316
> >             Project: Lucene - Java
> >          Issue Type: Bug
> >          Components: Query/Scoring
> >    Affects Versions: 2.3
> >         Environment: All
> >            Reporter: Todd Feak
> >            Assignee: Michael McCandless
> >            Priority: Minor
> >             Fix For: 2.9
> >
> >         Attachments: LUCENE-1316.patch, LUCENE-1316.patch,  
> LUCENE-1316.patch, LUCENE_1316.patch, LUCENE_1316.patch,  
> LUCENE_1316.patch, MatchAllDocsQuery.java
> >
> >   Original Estimate: 1h
> >  Remaining Estimate: 1h
> >
> > The isDeleted() method on IndexReader has been mentioned a number  
> of times as a potential synchronization bottleneck. However, the  
> reason this  bottleneck occurs is actually at a higher level that  
> wasn't focused on (at least in the threads I read).
> > In every case I saw where a stack trace was provided to show the  
> lock/block, higher in the stack you see the MatchAllScorer.next()  
> method. In Solr paricularly, this scorer is used for "NOT" queries.  
> We saw incredibly poor performance (order of magnitude) on our load  
> tests for NOT queries, due to this bottleneck. The problem is that  
> every single document is run through this isDeleted() method, which  
> is synchronized. Having an optimized index exacerbates this issues,  
> as there is only a single SegmentReader to synchronize on, causing a  
> major thread pileup waiting for the lock.
> > By simply having the MatchAllScorer see if there have been any  
> deletions in the reader, much of this can be avoided. Especially in  
> a read-only environment for production where you have slaves doing  
> all the high load searching.
> > I modified line 67 in the MatchAllDocsQuery
> > FROM:
> >   if (!reader.isDeleted(id)) {
> > TO:
> >   if (!reader.hasDeletions() || !reader.isDeleted(id)) {
> > In our micro load test for NOT queries only, this was a major  
> performance improvement.  We also got the same query results. I  
> don't believe this will improve the situation for indexes that have  
> deletions.
> > Please consider making this adjustment for a future bug fix release.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message