Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Date: Mon, 9 Jan 2017 18:15:58 +0000 (UTC)
From: "Hoss Man (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: <JIRA.13032749.1483752752000.687778.1483985758682@Atlassian.JIRA>
In-Reply-To: <JIRA.13032749.1483752752000@Atlassian.JIRA>
References: <JIRA.13032749.1483752752000@Atlassian.JIRA> <JIRA.13032749.1483752752576@arcas>
Subject: [jira] [Commented] (SOLR-9941) log replay redundently (pre-)applies
 DBQs as if they were out of order
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 09 Jan 2017 18:16:01 -0000


    [ https://issues.apache.org/jira/browse/SOLR-9941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812443#comment-15812443 ] 

Hoss Man commented on SOLR-9941:
--------------------------------

The patch looks pretty good to me -- but i'm not a tlog expert.

One question i have is: if the only code paths that call {{recoverFromLog(boolean)}} are "startup" paths that pass {{true}} why do we need the optional argument?  why not just refactor the method to always use the new logic?

> log replay redundently (pre-)applies DBQs as if they were out of order
> ----------------------------------------------------------------------
>
>                 Key: SOLR-9941
>                 URL: https://issues.apache.org/jira/browse/SOLR-9941
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Hoss Man
>         Attachments: SOLR-9941.patch, SOLR-9941.patch, SOLR-9941.patch
>
>
> There's kind of an odd situation that arises when a Solr node starts up (after a crash) and tries to recover from it's tlog that causes deletes to be redundantly & excessively applied -- at a minimum it causes confusing really log messages....
> * {{UpdateLog.init(...)}} creates {{TransactionLog}} instances for the most recent log files found (based on numRecordsToKeep) and then builds a {{RecentUpdates}} instance from them
> * Delete entries from the {{RecentUpdates}} are used to populate 2 lists:
> ** {{deleteByQueries}}
> ** {{oldDeletes}} (for deleteById).
> * Then when {{UpdateLog.recoverFromLog}} is called a {{LogReplayer}} is used to replay any (uncommited) {{TransactionLog}} enteries
> ** during replay {{UpdateLog}} delegates to the UpdateRequestProcessorChain to for the various adds/deletes, etc...
> ** when an add makes it to {{RunUpdateProcessor}} it delegates to {{DirectUpdateHandler2}}, which (independent of the fact that we're in log replay) calls {{UpdateLog.getDBQNewer}} for every add, looking for any "Reordered" deletes that have a version greater then the add
> *** if it finds _any_ DBQs "newer" then the document being added, it does a low level {{IndexWriter.updateDocument}} and then immediately executes _all_ the newer DBQs ... _once per add_
> ** these deletes are *also* still executed as part of the normal tlog replay, because they are in the tlog.
> Which means if you are recovering from a tlog with 90 addDocs, followed by 5 DBQs, then *each* of those 5 DBQs will each be executed 91 times -- and for 90 of those executions, a DUH2 INFO log messages will say {{"Reordered DBQs detected. ..."}} even tough the only reason they are out of order is because Solr is deliberately applying them out of order.
> * At a minimum we should improve the log messages
> * Ideally we should stop (pre-emptively) applying these deletes during tlog replay.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org