incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicholas Knight <nkni...@runawaynet.com>
Subject Re: What happens if there is a collision?
Date Fri, 22 Oct 2010 00:27:50 GMT

On Oct 22, 2010, at 7:41 AM, Jérôme Verstrynge wrote:

> Let's imagine that A initiates its column write at: 334450 ms with 'AAA' and timestamp
334450 ms
> Let's imagine that E initiates its column write at: 334451 ms with 'ZZZ'and timestamp
334450 ms
> (E is the latest write)
> 
> Let's imagine that A reaches C at 334455 ms and performs its write.
> Let's imagine that E reaches C at 334456 ms and attempts to performs its write. It will
loose the timestamp-tie ('AAA' is greater than 'ZZZ').


How is this any different from E's perspective than if A had come along a moment later with
timestamp 334452?

What you describe is an application in *desperate* need of either a serious redesign, or a
distributed locking mechanism.

This really isn't a Cassandra-specific problem, Cassandra just happens to be the distributed
storage system at issue. Any such system without a locking mechanism will present some form
of this problem, and the answer will be the same: Avoid it in the application design, or incorporate
a locking mechanism into the application.


> If there is a timestamp-tie, then the context becomes uncertain for E, out of the blue.
> If application E can't be sure about what has been saved in Cassandra, it cannot rely
on what it has in memory. It is a vicious circle. It can't anticipate on the potential actions
of A on the column too.


And how is this different from E's data being overwritten with a later timestamp? Either way,
what E thinks is in Cassandra really isn't.

If you need to make sure you have consistency at this level, you *need* a locking mechanism.


> This is unsual for any application, but may be this is the price to pay for using Cassandra.
Fair enough.


Hardly. Any non-serial application that doesn't use some form of locking has this exact same
problem at all levels of storage, possibly even in its internal variables.


> 
> If E is not informed of the timestamp tie, then it is left alone in the dark. Hence,
this is why I say Cassandra is not deterministic to E. The result of a write is potentially
non-deterministic in what it actually performs.


Cassandra is deterministic for a given input. What you're saying is you aren't properly controlling
the input that your application is giving it.


> If E was aware that it lost a timestamp-tie, it would know that there is a possible gap
between its internal memory representation and what it tried to save into Cassandra. That
is, EVEN if there is no further write on that same column (or, in other words, regardless
of any potential subsequent races).


What is the significance of this?


> 
> If E was informed it lost a timestamp-tie, it could re-read the column (and let's assume
that there is no further write in between, but this does not change anything to the argument).
It could spot that its write for timestamp value 334450 ms failed, and also the reason why
('AAA' greater than 'ZZZ). It could operate a new write, which eventually could result in
another timestamp-tie, but at least it would be informed about it too... It would have a safety
net.


To what end? A and E would apparently get into some sort of never-ending fight. The application
as described is broken and needs to be fixed.


> 
> The case I am trying to cover is the case where the context for application E becomes
invalid because of a successful write call to Cassandra without registration of 'ZZZ'. How
can Cassandra call it a successful write, when in fact, it isn't for application E? I believe
Cassandra should notify application E one way or another. This is why I mentioned an extra
timestamp-tie flag in the write ACK sent by nodes back to node E.


Here's part of the problem. You're seeing E as a distinct application from A which can behave
completely independently. You need to stop thinking like that. It leads to broken architectures

Even if the E and A processes come from entirely different code bases, you need to start by
thinking of them as one application. That application is broken.


> 
> The subsequent question I have is:
> 
> If 'value breaks timestamp-tie', how does Cassandra behave in case of updates? If there
is a column with value 'AAA' at 334450 ms and an application explicitely wants to update this
value to 'ZZZ' for 334450 ms, it seems like the timestamp-tie will prevent that. Hence, the
update/mutation would be undeterministic to E. It seems like one should first delete the existing
record and write a new one (and that could lead to race conditions and timestamp-ties too).


You need a locking mechanism. Timestamps aren't the droids you're looking for.


> I think this should be documented, because engineers will hit that 'local' undeterministic
issue for sure if two instances of their applications perform 'completed writes' in the same
column family. Completed does not mean successful, even with quorum (or ALL). They ought to
know it.

I'm honestly not sure why they wouldn't. One need only perform a very cursory investigation
of Cassandra to realize that addition of a locking mechanism is necessary for many applications,
such as the one described here.

-NK


Mime
View raw message