Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Wed, 5 Nov 2014 21:48:34 +0000 (UTC)
From: "Brandon Williams (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12753136.1415214613000.428016.1415224114491@Atlassian.JIRA>
In-Reply-To: <JIRA.12753136.1415214613000@Atlassian.JIRA>
References: <JIRA.12753136.1415214613000@Atlassian.JIRA>
 <JIRA.12753136.1415214613873@arcas>
Subject: [jira] [Updated] (CASSANDRA-8260) Replacing a node can leave the
 old node in system.peers on the replacement
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/CASSANDRA-8260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Brandon Williams updated CASSANDRA-8260:
----------------------------------------
    Attachment: 8260.txt

The most obvious solution here is to have a longer quarantine on the replacement node than its neighbors.  Patch to do this, and also protect us as soon as we finish the replacement, since in testing I found a small window where the extra digest can sneak in there too.

> Replacing a node can leave the old node in system.peers on the replacement
> --------------------------------------------------------------------------
>
>                 Key: CASSANDRA-8260
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8260
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Brandon Williams
>            Assignee: Brandon Williams
>             Fix For: 2.0.12
>
>         Attachments: 8260.txt
>
>
> Here's what happens:
> Nodes: X, Y, Z. Z replaces Y which is dead.
> 0. Replacement finishes
> 1. Z removes Y, quarantines and evicts (that is, removes the state)
> 2. X sees the replacement, quarantines, but keeps state
> 3. 60s elapses
> 4. quarantine on Z expires
> 5. X sends syn to Z, repopulates Y endpoint state and persists to system.peers, but Z sees the conflict and does not update tMD for Y. 
> 6. FatClient timer on Z starts counting.
> 7. quarantine on X expires, fat client has been idle, evicts and re-quarantines
> 8. 30s elapses
> 9. Fat client timeout occurs on Z, evicts and re-quarantines
> 10. 30s elapses
> 11. quarantine on X expires, so it never gets repopulated with Y since Z already removed it
> It's important to note here that there is a small but relevant gap between steps 1 and 2, which then correlates to steps 4 and 5, and step 5 is where the problem occurs. This also explains why it looks related to RING_DELAY, since the quarantine is RING_DELAY * 2, but Y never quarantines and the fat client timeout is RING_DELAY, effectively making the discrepancy near equal to RING_DELAY in the end.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)