Hi Mike,

Is this patch self contained or do I need the other patches from LUCENE-1316?
I tried just this patch, but I get a NPE in MultiSegment Reader (current  = null):

    at org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.doc(MultiSegmentReader.java:531)
    at org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.score(ValueSourceQuery.java:142)
    at org.apache.lucene.search.function.CustomScoreQuery$CustomScorer.score(CustomScoreQuery.java:389)
    at org.apache.lucene.search.Scorer.score(Scorer.java:58)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
    at org.apache.lucene.search.Searcher.search(Searcher.java:118)

Peter


On Fri, Feb 6, 2009 at 7:13 PM, Michael McCandless <lucene@mikemccandless.com> wrote:

OK I just put the patch on there -- let me know.  Thanks,

Mike


Peter Keegan wrote:

Thanks, Mike. I'll be happy to help test the patch.

Peter

On Fri, Feb 6, 2009 at 6:44 PM, Michael McCandless <lucene@mikemccandless.com> wrote:

OK I created https://issues.apache.org/jira/browse/LUCENE-1538.

Mike


Peter Keegan wrote:

I just ran into the same bottleneck with ValueSourceScorer.
Here's a stack trace:

Name: QueryThread group 1,#4
State: BLOCKED on org.apache.lucene.index.SegmentReader@49d7fb83 owned by: QueryThread group 1,#8
Total blocked: 1,535,881  Total waited: 1,080

Stack trace:
org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:663)
org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:221)
org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.next(ValueSourceQuery.java:138)
org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.skipTo(ValueSourceQuery.java:159)
org.apache.lucene.search.function.CustomScoreQuery$CustomScorer.skipTo(CustomScoreQuery.java:399)
org.apache.lucene.search.ConjunctionScorer.doNext(ConjunctionScorer.java:59)
org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:51)
org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
org.apache.lucene.search.Searcher.search(Searcher.java:118)

Peter


On Sun, Jan 25, 2009 at 9:41 AM, Michael McCandless (JIRA) <jira@apache.org> wrote:

  [ https://issues.apache.org/jira/browse/LUCENE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless resolved LUCENE-1316.
----------------------------------------

 Resolution: Fixed

Committed revision 737513.  Thanks everyone!

> Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer
> ------------------------------------------------------------------------
>
>                 Key: LUCENE-1316
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1316
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Query/Scoring
>    Affects Versions: 2.3
>         Environment: All
>            Reporter: Todd Feak
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 2.9
>
>         Attachments: LUCENE-1316.patch, LUCENE-1316.patch, LUCENE-1316.patch, LUCENE_1316.patch, LUCENE_1316.patch, LUCENE_1316.patch, MatchAllDocsQuery.java
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The isDeleted() method on IndexReader has been mentioned a number of times as a potential synchronization bottleneck. However, the reason this  bottleneck occurs is actually at a higher level that wasn't focused on (at least in the threads I read).
> In every case I saw where a stack trace was provided to show the lock/block, higher in the stack you see the MatchAllScorer.next() method. In Solr paricularly, this scorer is used for "NOT" queries. We saw incredibly poor performance (order of magnitude) on our load tests for NOT queries, due to this bottleneck. The problem is that every single document is run through this isDeleted() method, which is synchronized. Having an optimized index exacerbates this issues, as there is only a single SegmentReader to synchronize on, causing a major thread pileup waiting for the lock.
> By simply having the MatchAllScorer see if there have been any deletions in the reader, much of this can be avoided. Especially in a read-only environment for production where you have slaves doing all the high load searching.
> I modified line 67 in the MatchAllDocsQuery
> FROM:
>   if (!reader.isDeleted(id)) {
> TO:
>   if (!reader.hasDeletions() || !reader.isDeleted(id)) {
> In our micro load test for NOT queries only, this was a major performance improvement.  We also got the same query results. I don't believe this will improve the situation for indexes that have deletions.
> Please consider making this adjustment for a future bug fix release.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org