lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Keegan <peterlkee...@gmail.com>
Subject Re: [jira] Resolved: (LUCENE-1316) Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer
Date Sat, 07 Feb 2009 20:56:55 GMT
Ok, I applied LUCENE-1316. (I had to patch MutliSegmentReader manually,
probably because I'm running 2.3.2 and patch is for 2.4?)
The patch fixes the sychronization bottleneck in ValueSourceQuery.

Thank you Mike!

Peter

On Sat, Feb 7, 2009 at 10:38 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:

>
> Hi Peter,
>
> You'll definitely need to start from the full patch in LUCENE-1316, which
> fixes SegmentReader & Multi*Reader to use AllTermDocs when given a null
> Term.
>
> Mike
>
>
> Peter Keegan wrote:
>
>  Hi Mike,
>>
>> Is this patch self contained or do I need the other patches from
>> LUCENE-1316?
>> I tried just this patch, but I get a NPE in MultiSegment Reader (current
>>  = null):
>>
>>    at
>> org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.doc(MultiSegmentReader.java:531)
>>    at
>> org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.score(ValueSourceQuery.java:142)
>>    at
>> org.apache.lucene.search.function.CustomScoreQuery$CustomScorer.score(CustomScoreQuery.java:389)
>>    at org.apache.lucene.search.Scorer.score(Scorer.java:58)
>>    at
>> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
>>    at org.apache.lucene.search.Searcher.search(Searcher.java:118)
>>
>> Peter
>>
>>
>> On Fri, Feb 6, 2009 at 7:13 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>> OK I just put the patch on there -- let me know.  Thanks,
>>
>> Mike
>>
>>
>> Peter Keegan wrote:
>>
>> Thanks, Mike. I'll be happy to help test the patch.
>>
>> Peter
>>
>> On Fri, Feb 6, 2009 at 6:44 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>> OK I created https://issues.apache.org/jira/browse/LUCENE-1538.
>>
>> Mike
>>
>>
>> Peter Keegan wrote:
>>
>> I just ran into the same bottleneck with ValueSourceScorer.
>> Here's a stack trace:
>>
>> Name: QueryThread group 1,#4
>> State: BLOCKED on org.apache.lucene.index.SegmentReader@49d7fb83 owned
>> by: QueryThread group 1,#8
>> Total blocked: 1,535,881  Total waited: 1,080
>>
>> Stack trace:
>> org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:663)
>> org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:221)
>>
>> org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.next(ValueSourceQuery.java:138)
>>
>> org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.skipTo(ValueSourceQuery.java:159)
>>
>> org.apache.lucene.search.function.CustomScoreQuery$CustomScorer.skipTo(CustomScoreQuery.java:399)
>>
>> org.apache.lucene.search.ConjunctionScorer.doNext(ConjunctionScorer.java:59)
>> org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:51)
>> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
>> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
>> org.apache.lucene.search.Searcher.search(Searcher.java:118)
>>
>> Peter
>>
>>
>> On Sun, Jan 25, 2009 at 9:41 AM, Michael McCandless (JIRA) <
>> jira@apache.org> wrote:
>>
>>  [
>> https://issues.apache.org/jira/browse/LUCENE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
>>  ]
>>
>> Michael McCandless resolved LUCENE-1316.
>> ----------------------------------------
>>
>>  Resolution: Fixed
>>
>> Committed revision 737513.  Thanks everyone!
>>
>> > Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer
>> > ------------------------------------------------------------------------
>> >
>> >                 Key: LUCENE-1316
>> >                 URL: https://issues.apache.org/jira/browse/LUCENE-1316
>> >             Project: Lucene - Java
>> >          Issue Type: Bug
>> >          Components: Query/Scoring
>> >    Affects Versions: 2.3
>> >         Environment: All
>> >            Reporter: Todd Feak
>> >            Assignee: Michael McCandless
>> >            Priority: Minor
>> >             Fix For: 2.9
>> >
>> >         Attachments: LUCENE-1316.patch, LUCENE-1316.patch,
>> LUCENE-1316.patch, LUCENE_1316.patch, LUCENE_1316.patch, LUCENE_1316.patch,
>> MatchAllDocsQuery.java
>> >
>> >   Original Estimate: 1h
>> >  Remaining Estimate: 1h
>> >
>> > The isDeleted() method on IndexReader has been mentioned a number of
>> times as a potential synchronization bottleneck. However, the reason this
>>  bottleneck occurs is actually at a higher level that wasn't focused on (at
>> least in the threads I read).
>> > In every case I saw where a stack trace was provided to show the
>> lock/block, higher in the stack you see the MatchAllScorer.next() method. In
>> Solr paricularly, this scorer is used for "NOT" queries. We saw incredibly
>> poor performance (order of magnitude) on our load tests for NOT queries, due
>> to this bottleneck. The problem is that every single document is run through
>> this isDeleted() method, which is synchronized. Having an optimized index
>> exacerbates this issues, as there is only a single SegmentReader to
>> synchronize on, causing a major thread pileup waiting for the lock.
>> > By simply having the MatchAllScorer see if there have been any deletions
>> in the reader, much of this can be avoided. Especially in a read-only
>> environment for production where you have slaves doing all the high load
>> searching.
>> > I modified line 67 in the MatchAllDocsQuery
>> > FROM:
>> >   if (!reader.isDeleted(id)) {
>> > TO:
>> >   if (!reader.hasDeletions() || !reader.isDeleted(id)) {
>> > In our micro load test for NOT queries only, this was a major
>> performance improvement.  We also got the same query results. I don't
>> believe this will improve the situation for indexes that have deletions.
>> > Please consider making this adjustment for a future bug fix release.
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> You can reply to this email to add a comment to the issue online.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

Mime
View raw message