Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Message-Id: <CF1DB614-A823-407B-B359-18CB22676B61@mikemccandless.com>
From: Michael McCandless <lucene@mikemccandless.com>
To: java-dev@lucene.apache.org
In-Reply-To: <e994873a0902070648w3331492r8d816ddd942a7a30@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Re: [jira] Resolved: (LUCENE-1316) Avoidable synchronization
  bottleneck in MatchAlldocsQuery$MatchAllScorer
Date: Sat, 7 Feb 2009 10:38:12 -0500
References: <545915468.1214409524974.JavaMail.jira@brutus>
 <276167189.1232894460193.JavaMail.jira@brutus>
 <e994873a0902061527s53549dcciae0f9162005a7076@mail.gmail.com>
 <048947B9-AB7B-4B33-AA9D-F0CDFFF77413@mikemccandless.com>
 <e994873a0902061608j5db9696ft58b6d9f9560e9f20@mail.gmail.com>
 <F1EC71A4-CE45-44DE-BA70-11ED01C040F7@mikemccandless.com>
 <e994873a0902070648w3331492r8d816ddd942a7a30@mail.gmail.com>


Hi Peter,

You'll definitely need to start from the full patch in LUCENE-1316,  
which fixes SegmentReader & Multi*Reader to use AllTermDocs when given  
a null Term.

Mike

Peter Keegan wrote:

> Hi Mike,
>
> Is this patch self contained or do I need the other patches from  
> LUCENE-1316?
> I tried just this patch, but I get a NPE in MultiSegment Reader  
> (current  = null):
>
>     at org.apache.lucene.index.MultiSegmentReader 
> $MultiTermDocs.doc(MultiSegmentReader.java:531)
>     at org.apache.lucene.search.function.ValueSourceQuery 
> $ValueSourceScorer.score(ValueSourceQuery.java:142)
>     at org.apache.lucene.search.function.CustomScoreQuery 
> $CustomScorer.score(CustomScoreQuery.java:389)
>     at org.apache.lucene.search.Scorer.score(Scorer.java:58)
>     at  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
>     at org.apache.lucene.search.Searcher.search(Searcher.java:118)
>
> Peter
>
>
> On Fri, Feb 6, 2009 at 7:13 PM, Michael McCandless <lucene@mikemccandless.com 
> > wrote:
>
> OK I just put the patch on there -- let me know.  Thanks,
>
> Mike
>
>
> Peter Keegan wrote:
>
> Thanks, Mike. I'll be happy to help test the patch.
>
> Peter
>
> On Fri, Feb 6, 2009 at 6:44 PM, Michael McCandless <lucene@mikemccandless.com 
> > wrote:
>
> OK I created https://issues.apache.org/jira/browse/LUCENE-1538.
>
> Mike
>
>
> Peter Keegan wrote:
>
> I just ran into the same bottleneck with ValueSourceScorer.
> Here's a stack trace:
>
> Name: QueryThread group 1,#4
> State: BLOCKED on org.apache.lucene.index.SegmentReader@49d7fb83  
> owned by: QueryThread group 1,#8
> Total blocked: 1,535,881  Total waited: 1,080
>
> Stack trace:
> org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java: 
> 663)
> org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:221)
> org.apache.lucene.search.function.ValueSourceQuery 
> $ValueSourceScorer.next(ValueSourceQuery.java:138)
> org.apache.lucene.search.function.ValueSourceQuery 
> $ValueSourceScorer.skipTo(ValueSourceQuery.java:159)
> org.apache.lucene.search.function.CustomScoreQuery 
> $CustomScorer.skipTo(CustomScoreQuery.java:399)
> org 
> .apache 
> .lucene.search.ConjunctionScorer.doNext(ConjunctionScorer.java:59)
> org 
> .apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java: 
> 51)
> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
> org.apache.lucene.search.Searcher.search(Searcher.java:118)
>
> Peter
>
>
> On Sun, Jan 25, 2009 at 9:41 AM, Michael McCandless (JIRA) <jira@apache.org 
> > wrote:
>
>   [ https://issues.apache.org/jira/browse/LUCENE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel 
>  ]
>
> Michael McCandless resolved LUCENE-1316.
> ----------------------------------------
>
>  Resolution: Fixed
>
> Committed revision 737513.  Thanks everyone!
>
> > Avoidable synchronization bottleneck in MatchAlldocsQuery 
> $MatchAllScorer
> >  
> ------------------------------------------------------------------------
> >
> >                 Key: LUCENE-1316
> >                 URL: https://issues.apache.org/jira/browse/LUCENE-1316
> >             Project: Lucene - Java
> >          Issue Type: Bug
> >          Components: Query/Scoring
> >    Affects Versions: 2.3
> >         Environment: All
> >            Reporter: Todd Feak
> >            Assignee: Michael McCandless
> >            Priority: Minor
> >             Fix For: 2.9
> >
> >         Attachments: LUCENE-1316.patch, LUCENE-1316.patch,  
> LUCENE-1316.patch, LUCENE_1316.patch, LUCENE_1316.patch,  
> LUCENE_1316.patch, MatchAllDocsQuery.java
> >
> >   Original Estimate: 1h
> >  Remaining Estimate: 1h
> >
> > The isDeleted() method on IndexReader has been mentioned a number  
> of times as a potential synchronization bottleneck. However, the  
> reason this  bottleneck occurs is actually at a higher level that  
> wasn't focused on (at least in the threads I read).
> > In every case I saw where a stack trace was provided to show the  
> lock/block, higher in the stack you see the MatchAllScorer.next()  
> method. In Solr paricularly, this scorer is used for "NOT" queries.  
> We saw incredibly poor performance (order of magnitude) on our load  
> tests for NOT queries, due to this bottleneck. The problem is that  
> every single document is run through this isDeleted() method, which  
> is synchronized. Having an optimized index exacerbates this issues,  
> as there is only a single SegmentReader to synchronize on, causing a  
> major thread pileup waiting for the lock.
> > By simply having the MatchAllScorer see if there have been any  
> deletions in the reader, much of this can be avoided. Especially in  
> a read-only environment for production where you have slaves doing  
> all the high load searching.
> > I modified line 67 in the MatchAllDocsQuery
> > FROM:
> >   if (!reader.isDeleted(id)) {
> > TO:
> >   if (!reader.hasDeletions() || !reader.isDeleted(id)) {
> > In our micro load test for NOT queries only, this was a major  
> performance improvement.  We also got the same query results. I  
> don't believe this will improve the situation for indexes that have  
> deletions.
> > Please consider making this adjustment for a future bug fix release.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org