lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Keegan <peterlkee...@gmail.com>
Subject Re: [jira] Resolved: (LUCENE-1316) Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer
Date Sat, 07 Feb 2009 00:08:37 GMT
Thanks, Mike. I'll be happy to help test the patch.

Peter

On Fri, Feb 6, 2009 at 6:44 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:

>
> OK I created https://issues.apache.org/jira/browse/LUCENE-1538.
>
> Mike
>
>
> Peter Keegan wrote:
>
>  I just ran into the same bottleneck with ValueSourceScorer.
>> Here's a stack trace:
>>
>> Name: QueryThread group 1,#4
>> State: BLOCKED on org.apache.lucene.index.SegmentReader@49d7fb83 owned
>> by: QueryThread group 1,#8
>> Total blocked: 1,535,881  Total waited: 1,080
>>
>> Stack trace:
>> org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:663)
>> org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:221)
>>
>> org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.next(ValueSourceQuery.java:138)
>>
>> org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.skipTo(ValueSourceQuery.java:159)
>>
>> org.apache.lucene.search.function.CustomScoreQuery$CustomScorer.skipTo(CustomScoreQuery.java:399)
>>
>> org.apache.lucene.search.ConjunctionScorer.doNext(ConjunctionScorer.java:59)
>> org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:51)
>> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
>> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
>> org.apache.lucene.search.Searcher.search(Searcher.java:118)
>>
>> Peter
>>
>>
>> On Sun, Jan 25, 2009 at 9:41 AM, Michael McCandless (JIRA) <
>> jira@apache.org> wrote:
>>
>>    [
>> https://issues.apache.org/jira/browse/LUCENE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
>>  ]
>>
>> Michael McCandless resolved LUCENE-1316.
>> ----------------------------------------
>>
>>   Resolution: Fixed
>>
>> Committed revision 737513.  Thanks everyone!
>>
>> > Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer
>> > ------------------------------------------------------------------------
>> >
>> >                 Key: LUCENE-1316
>> >                 URL: https://issues.apache.org/jira/browse/LUCENE-1316
>> >             Project: Lucene - Java
>> >          Issue Type: Bug
>> >          Components: Query/Scoring
>> >    Affects Versions: 2.3
>> >         Environment: All
>> >            Reporter: Todd Feak
>> >            Assignee: Michael McCandless
>> >            Priority: Minor
>> >             Fix For: 2.9
>> >
>> >         Attachments: LUCENE-1316.patch, LUCENE-1316.patch,
>> LUCENE-1316.patch, LUCENE_1316.patch, LUCENE_1316.patch, LUCENE_1316.patch,
>> MatchAllDocsQuery.java
>> >
>> >   Original Estimate: 1h
>> >  Remaining Estimate: 1h
>> >
>> > The isDeleted() method on IndexReader has been mentioned a number of
>> times as a potential synchronization bottleneck. However, the reason this
>>  bottleneck occurs is actually at a higher level that wasn't focused on (at
>> least in the threads I read).
>> > In every case I saw where a stack trace was provided to show the
>> lock/block, higher in the stack you see the MatchAllScorer.next() method. In
>> Solr paricularly, this scorer is used for "NOT" queries. We saw incredibly
>> poor performance (order of magnitude) on our load tests for NOT queries, due
>> to this bottleneck. The problem is that every single document is run through
>> this isDeleted() method, which is synchronized. Having an optimized index
>> exacerbates this issues, as there is only a single SegmentReader to
>> synchronize on, causing a major thread pileup waiting for the lock.
>> > By simply having the MatchAllScorer see if there have been any deletions
>> in the reader, much of this can be avoided. Especially in a read-only
>> environment for production where you have slaves doing all the high load
>> searching.
>> > I modified line 67 in the MatchAllDocsQuery
>> > FROM:
>> >   if (!reader.isDeleted(id)) {
>> > TO:
>> >   if (!reader.hasDeletions() || !reader.isDeleted(id)) {
>> > In our micro load test for NOT queries only, this was a major
>> performance improvement.  We also got the same query results. I don't
>> believe this will improve the situation for indexes that have deletions.
>> > Please consider making this adjustment for a future bug fix release.
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> You can reply to this email to add a comment to the issue online.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

Mime
View raw message