cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Stupp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-9649) Paxos ballot in StorageProxy could clash
Date Thu, 25 Jun 2015 14:22:05 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-9649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14601256#comment-14601256
] 

Robert Stupp commented on CASSANDRA-9649:
-----------------------------------------

[~slebresne], I wanted to ask why we don't use ┬Ás precision here - answered while lurking
to your branch. The changes LGTM.
The main problem (_dear operator, do never ever set the wrong system time_) is indeed a big
one - a problem that we can detect and react to these situations:
# check whether the system time of a starting/joining node is within some allowed range and
prevent it to join the ring
# detect abrupt/big clock skews of a node via gossip and issue warnings
# detect abrupt/big clock skews by comparing the diff of {{System.currentTimeMillis()}} and
{{nanoTime()}} and issue warnings

The consequences of an abrupt/big clock skew could be mitigated by de-coupling {{System.currentTimeMillis()}}
from {{UUIDGen}} and {{ClientState}} and only allow it to go faster/slower at a max rate -
basically preventing an _immediate_ system clock change from 06/25/2015-16:18:30 to 06/25/2015-16:48:30
(what NTP usually does, advance slowly) - or even do not allow the "internal clock" to ever
go backwards. WDYT? (I think there was some discussion around that a while ago - but honestly
I cannot recall.)

> Paxos ballot in StorageProxy could clash
> ----------------------------------------
>
>                 Key: CASSANDRA-9649
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9649
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Stefania
>            Assignee: Stefania
>            Priority: Minor
>
> This code in {{StorageProxy.beginAndRepairPaxos()}} takes a timestamp in microseconds
but divides it by 1000 before adding one. So if the summary is null, ballotMillis would be
the same for up to 1000 possible state timestamp values:
> {code}
>     long currentTime = (state.getTimestamp() / 1000) + 1;
>     long ballotMillis = summary == null
>                                  ? currentTime
>                                  : Math.max(currentTime, 1 +    UUIDGen.unixTimestamp(summary.mostRecentInProgressCommit.ballot));
>     UUID ballot = UUIDGen.getTimeUUID(ballotMillis);
> {code}
> {{state.getTimestamp()}} returns the time in micro seconds and it ensures to add one
microsecond to any previously used timestamp if the client sends the same or an older timestamp.

> Initially I used this code in {{ModificationStatement.casInternal()}}, introduced by
CASSANDRA-9160 to support cas unit tests, but occasionally these tests were failing. It was
only when I ensured uniqueness of the ballot that the tests started to pass reliably.
> I wonder if we could ever have the same issue in StorageProxy?
> cc [~jbellis] and [~slebresne] for CASSANDRA-7801



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message