cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brandon Williams (JIRA)" <>
Subject [jira] [Updated] (CASSANDRA-8260) Replacing a node can leave the old node in system.peers on the replacement
Date Wed, 05 Nov 2014 21:48:34 GMT


Brandon Williams updated CASSANDRA-8260:
    Attachment: 8260.txt

The most obvious solution here is to have a longer quarantine on the replacement node than
its neighbors.  Patch to do this, and also protect us as soon as we finish the replacement,
since in testing I found a small window where the extra digest can sneak in there too.

> Replacing a node can leave the old node in system.peers on the replacement
> --------------------------------------------------------------------------
>                 Key: CASSANDRA-8260
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Brandon Williams
>            Assignee: Brandon Williams
>             Fix For: 2.0.12
>         Attachments: 8260.txt
> Here's what happens:
> Nodes: X, Y, Z. Z replaces Y which is dead.
> 0. Replacement finishes
> 1. Z removes Y, quarantines and evicts (that is, removes the state)
> 2. X sees the replacement, quarantines, but keeps state
> 3. 60s elapses
> 4. quarantine on Z expires
> 5. X sends syn to Z, repopulates Y endpoint state and persists to system.peers, but Z
sees the conflict and does not update tMD for Y. 
> 6. FatClient timer on Z starts counting.
> 7. quarantine on X expires, fat client has been idle, evicts and re-quarantines
> 8. 30s elapses
> 9. Fat client timeout occurs on Z, evicts and re-quarantines
> 10. 30s elapses
> 11. quarantine on X expires, so it never gets repopulated with Y since Z already removed
> It's important to note here that there is a small but relevant gap between steps 1 and
2, which then correlates to steps 4 and 5, and step 5 is where the problem occurs. This also
explains why it looks related to RING_DELAY, since the quarantine is RING_DELAY * 2, but Y
never quarantines and the fat client timeout is RING_DELAY, effectively making the discrepancy
near equal to RING_DELAY in the end.

This message was sent by Atlassian JIRA

View raw message