cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From wool...@gmail.com
Subject Re: Is Cassandra really Strong consistency?
Date Mon, 07 Sep 2015 10:59:33 GMT

It you need strong consistency and don't mind lower transaction rate, you're better off with
base 

Sent from my iPhone

> On Sep 7, 2015, at 5:55 AM, ibrahim El-sanosi <ibrahimsabattt@gmail.com> wrote:
> 
> Ok,
> 
>  
> 
> With LWT, I am completely understand that it will achieve total order/linearizability,
therefore above scenario cannot occur.
> 
> 
> 
> However, when you said "the scenario  will occur if your clocks are not sync’d". This
is ambiguous statement. Because both client and server sides are likely to have different
wall-clock for many reasons:
> 
> 1. Clients and Cassandra server are located in different regions, resulting in different
timestamps.
> 
> 2. Among Cassandra servers (or replicas) is also possible to have different timestamps,
so above scenario can occur. Even we use NTP to synchronize the clock, the scenario can happen
at least the different in milliseconds. 
> 
> 
> 
> That means that the timestamps for writes are derived either from a single Cassandra
server clock, or a single app server clock. These clocks can flow backwards.
> 
> 
> 
> What do you think?
> 
>  
> 
> Ibrahim
> 
> 
>> On Mon, Sep 7, 2015 at 1:26 AM, Jeff Jirsa <jeff.jirsa@crowdstrike.com> wrote:
>> 
>> Yes, your scenario can occur, and will occur if your clocks are not sync’d.
>> 
>> Either you sync your clocks to appropriate tolerances, or you don’t write without
checking the existing value (with LWT). There is no other resolution in cassandra – there
are no vector clocks to allow you to manage the conflict on your own at this point.  
>> 
>> 
>> From: ibrahim El-sanosi
>> Reply-To: "user@cassandra.apache.org"
>> Date: Sunday, September 6, 2015 at 11:28 AM
>> 
>> To: "user@cassandra.apache.org"
>> Subject: Re: Is Cassandra really Strong consistency?
>> 
>>  
>>  
>> Yes, LWT is another case and different compared to what my scenario is about. I am
not talking about LWT and CAS, it is true that LWT uses logical clock by utilising Paxos.
But my scenario is talking about using timestamp and Last-Write-Wins.
>>  
>>  
>> If anyone can read the above scenario and confirm whether this can occur or not,
if it is possible, how current Cassandra can solve it?
>>  
>> Regards,
>>  
>> Ibrahim
>>  
>>  
>> If anyone can read the above scenario and confirm whether this can occur or not,
if it is possible, how current Cassandra can solve it?
>>  
>> Regards,
>>  
>> Ibrahim
>> 
>>> On Sun, Sep 6, 2015 at 5:57 PM, Jeff Jirsa <jeff.jirsa@crowdstrike.com>
wrote:
>>> In the cases where NTP and client timestamps with microsecond resolution is insufficient,
LWT “IF EXISTS, IF NOT EXISTS” is generally used. 
>>> 
>>> 
>>> From: ibrahim El-sanosi
>>> Reply-To: "user@cassandra.apache.org"
>>> Date: Sunday, September 6, 2015 at 7:40 AM
>>> To: "user@cassandra.apache.org"
>>> 
>>> Subject: Re: Is Cassandra really Strong consistency?
>>> 
>>>  
>>> I have done some research about “timestamps could jump back and forth arbitrarily
if you talk to different nodes”.
>>> To summarise,  it is possible in Cassandra for following scenario can happen
in sequence:
>>>  
>>> Process A writes w1 with timestamp t=2
>>> Process B reads w1
>>> Process B writes w2 with timestamp t=1
>>> Process B reads w1, but expected w2
>>> If the system clock goes backwards for any reason, Cassandra’s session consistency
guarantees no longer hold, even consistency level is write/read CL = QOURUM  or write CL =
ALL and read CL =one.
>>>  
>>> Moreover, even we use NTP, the problem above can occur. That means that the timestamps
for writes are derived either from a single Cassandra server clock, or a single app server
clock. These clocks can flow backwards, for a number of “reasons”:
>>> Hardware wonkiness can push clocks days or centuries into the future or past.
>>> Virtualization can wreak havoc on kernel timekeeping.
>>> Misconfigured nodes may not have NTP enabled, or may not be able to reach upstream
sources.
>>> Upstream NTP servers can lie.
>>> When the problem is identified and fixed, NTP corrects large time differentials
by jumping the clock discontinously to the correct time.
>>> Even when perfectly synchronized, POSIX time itself is not monotonic.
>>>  
>>> If you want to read more this link can give you a lot hints.
>>>  
>>> Regards,
>>>  
>>> Ibrahim
>>> 
>>>> On Sun, Sep 6, 2015 at 2:01 PM, Edouard COLE <Edouard.COLE@rgsystem.com>
wrote:
>>>> @ibrahim: When saying "clocks should be synchronized", it includes Cassandra
nodes AND clients
>>>> 
>>>> NTP is the way to go
>>>> 
>>>>> Le 6 sept. 2015 à 14:56, Laing, Michael <michael.laing@nytimes.com>
a écrit :
>>>>> 
>>>>> https://en.wikipedia.org/wiki/Network_Time_Protocol
>>>>> 
>>>>>> On Sun, Sep 6, 2015 at 8:23 AM, ibrahim El-sanosi <ibrahimsabattt@gmail.com>
wrote:
>>>>>> Assume the Cassandra cluster is located in somewhere in US. Clients
that connect from different part of the world will have different timestamp (if we rely on
client timestamp to store write) or If a coordinator is responsible for generating timestamp
during the write, it also may have different time among replicas, resulting in write conflict
can occur and impossible to resolve.
>>>>>>  
>>>>>> When you are saying “Clocks should be synchronized”, does Cassandra
synchronize the clock if so how can you refer me to any related article?
>>>>>>  
>>>>>> Regards,
>>>>>>  
>>>>>> Ibrahim
>>>>>> 
>>>>>>> On Sun, Sep 6, 2015 at 1:23 PM, Daniel Schulz <danielschulz2005@hotmail.com>
wrote:
>>>>>>> Cassandra is not changing clock settings; it does use it to omit
TTL'ed rows in compaction phases. So make sure your nodes agree on the very same time using
e.g. NTP. It is very crucial for data integrity on most distributed systems.
>>>>>>> 
>>>>>>> Date: Sun, 6 Sep 2015 13:10:14 +0100
>>>>>>> Subject: Re: Is Cassandra really Strong consistency?
>>>>>>> From: ibrahimsabattt@gmail.com
>>>>>>> To: user@cassandra.apache.org
>>>>>>> 
>>>>>>> 
>>>>>>> Do you mean Cassandra does synchronize the clock across all the
cluster, if yes how it does so, or could you refer me to any related article?
>>>>>>> 
>>>>>>> Thank you
>>>>>>> 
>>>>>>> 
>>>>>>> Ibrahim
>>>>>>> 
>>>>>>> On Sun, Sep 6, 2015 at 1:00 PM, Laing, Michael <michael.laing@nytimes.com>
wrote:
>>>>>>> I think I saw this before.
>>>>>>> 
>>>>>>> Clocks must be synchronized.
>>>>>>> 
>>>>>>> On Sun, Sep 6, 2015 at 7:28 AM, ibrahim El-sanosi <ibrahimsabattt@gmail.com>
wrote:
>>>>>>> Hi folks,
>>>>>>> 
>>>>>>> Assume we have 4-nodes cluster N1, N2, N3, and N4 and replication
factor is 3.  When write CL =ALL and read CL=ONE:
>>>>>>> 
>>>>>>> Client c1 sends W1 = [k1,V1] to N1 (a coordinator).  A coordinator
(N1) generates timestamp Mon 05-09-2015 11:30:40,200 (according to its local clock) and assigned
 it to W1 and sends the W1 to N2, N3, and N4. After few seconds, Client c2 sends W2 = [K1,
V2] to N4 (a coordinator). A coordinator (N4) generates timestamp Mon 05-09-2015 11:30:38,200
(according to its local clock, but assume here N4 clock a bit behind, nearly 2 seconds) and
assigned it to W2 and sends the W2 to N2, N3, and N4 (itself). 
>>>>>>> 
>>>>>>> As we have write CL =ALL and read CL = ONE. Now, Client c2 wants
to read K1, connects to a coordinator N1, a coordinator sends read K1 to N2, picking latest
timestamp which is [K1, V1]:Mon 05-09-2015 11:30:40,200.
>>>>>>> 
>>>>>>> So in this scenario, the latest data that wrote to the replicas
is [K1, V2] which should be the correct one, but it reads [K1,V1] because of divert clock.

>>>>>>> 
>>>>>>> 
>>>>>>> Can such scenario occur?
>>>>>>> 
>>>>>>> Thank you
>>>>>>> 
> 

Mime
View raw message