lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Feak (JIRA)" <>
Subject [jira] Created: (LUCENE-1316) Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer
Date Wed, 25 Jun 2008 15:58:44 GMT
Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer

                 Key: LUCENE-1316
             Project: Lucene - Java
          Issue Type: Bug
          Components: Query/Scoring
    Affects Versions: 2.3
         Environment: All
            Reporter: Todd Feak
            Priority: Minor

The isDeleted() method on IndexReader has been mentioned a number of times as a potential
synchronization bottleneck. However, the reason this  bottleneck occurs is actually at a higher
level that wasn't focused on (at least in the threads I read).

In every case I saw where a stack trace was provided to show the lock/block, higher in the
stack you see the method. In Solr paricularly, this scorer is used for
"NOT" queries. We saw incredibly poor performance (order of magnitude) on our load tests for
NOT queries, due to this bottleneck. The problem is that every single document is run through
this isDeleted() method, which is synchronized. Having an optimized index exacerbates this
issues, as there is only a single SegmentReader to synchronize on, causing a major thread
pileup waiting for the lock.

By simply having the MatchAllScorer see if there have been any deletions in the reader, much
of this can be avoided. Especially in a read-only environment for production where you have
slaves doing all the high load searching.

I modified line 67 in the MatchAllDocsQuery
  if (!reader.isDeleted(id)) {
  if (!reader.hasDeletions() || !reader.isDeleted(id)) {

In our micro load test for NOT queries only, this was a major performance improvement.  We
also got the same query results. I don't believe this will improve the situation for indexes
that have deletions. 

Please consider making this adjustment for a future bug fix release.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message