From user-return-32434-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Wed Mar 6 07:02:14 2013 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 68509EADD for ; Wed, 6 Mar 2013 07:02:14 +0000 (UTC) Received: (qmail 60823 invoked by uid 500); 6 Mar 2013 07:02:11 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 60495 invoked by uid 500); 6 Mar 2013 07:02:10 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 60485 invoked by uid 99); 6 Mar 2013 07:02:10 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Mar 2013 07:02:10 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a79.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Mar 2013 07:02:05 +0000 Received: from homiemail-a79.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a79.g.dreamhost.com (Postfix) with ESMTP id 87F4B7D406E for ; Tue, 5 Mar 2013 23:01:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=oZ5jn5XqzthnmEtpDw5TBSVq7C w=; b=v3YX4KU8cDuWU2pBItuTJRi0AJyVuvisn80jKMUIsgg98qvZt+uhsS/fdq O5G31dTezmIdIigYSCTYruDlUfoj2ELbFkQnOiWdiDlk9vWGTM7KE9Y7wqKALdS8 +P0DaOjWHW6T6+1k+CYF/Pz+0RnXDzSwJo8ijyuFm8I9AsDIA= Received: from [192.168.168.119] (c-98-234-52-29.hsd1.ca.comcast.net [98.234.52.29]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a79.g.dreamhost.com (Postfix) with ESMTPSA id 565737D4059 for ; Tue, 5 Mar 2013 23:01:24 -0800 (PST) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_19799891-0A0E-4A7C-9476-4A9185FF137F" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Consistent problem when solve Digest mismatch Date: Tue, 5 Mar 2013 23:01:43 -0800 References: To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1499) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_19799891-0A0E-4A7C-9476-4A9185FF137F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 > Otherwise, it means the version conflict solving strong depends on = global sequence id (timestamp) which need provide by client ? Yes.=20 If you have an area of your data model that has a high degree of = concurrency C* may not be the right match. In 1.1 we have atomic updates so clients see either the entire write or = none of it. And sometimes you can design a data model that does mutate = shared values, but writes ledger entries instead. See Matt Denis talk = here http://www.datastax.com/events/cassandrasummit2012/presentations or = this post http://thelastpickle.com/2012/08/18/Sorting-Lists-For-Humans/ Cheers ----------------- Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 4/03/2013, at 4:30 PM, Jason Tang wrote: > Hi=20 >=20 > The timestamp provided by my client is unix timestamp (with ntp), and = as I said, due to the ntp drift, the local unix timestamp is not = accurately synchronized (compare to my case). >=20 > So for short, client can not provide global sequence number to = indicate the event order. >=20 > But I wonder, I configured Cassandra consistency level as write = QUORUM. So for one record, I suppose Cassandra has the ability to decide = the final update results. >=20 > Otherwise, it means the version conflict solving strong depends on = global sequence id (timestamp) which need provide by client ? >=20 >=20 > //Tang >=20 >=20 > 2013/3/4 Sylvain Lebresne > The problem is, what is the sequence number you are talking about is = exactly? >=20 > Or let me put it another way: if you do have a sequence number that = provides a total ordering of your operation, then that is exactly what = you should use as your timestamp. What Cassandra calls the timestamp, is = exactly what you call seqID, it's the number Cassandra uses to decide = the order of operation. >=20 > Except that in real life, provided you have more than one client = talking to Cassandra, then providing a total ordering of operation is = hard, and in fact not doable efficiently. So in practice, people use = unix timestamp (with ntp) which provide a very good while cheap = approximation of the real life order of operations. >=20 > But again, if you do know how to assign a more precise "timestamp", = Cassandra let you use that: you can provid your own timestamp (using = unix timestamp is just the default). The point being, unix timestamp is = the better approximation we have in practice. >=20 > -- > Sylvain >=20 >=20 > On Mon, Mar 4, 2013 at 9:26 AM, Jason Tang = wrote: > Hi >=20 > Previous I met a consistency problem, you can refer the link below = for the whole story. > = http://mail-archives.apache.org/mod_mbox/cassandra-user/201206.mbox/%3CCAF= b+LUxna0jiY0V=3DAvXKzUdxSjApYm4zWk=3DKa9LJM-txc04Gjw@mail.gmail.com%3E >=20 > And after check the code, seems I found some clue of the problem. = Maybe some one can check this. >=20 > For short, I have Cassandra cluster (1.0.3), The consistency level = is read/write quorum, replication_factor is 3.=20 >=20 > Here is event sequence: >=20 > seqID NodeA NodeB NodeC > 1. New New New > 2. Update Update Update > 3. Delete Delete =20 >=20 > When try to read from NodeB and NodeC, "Digest mismatch" exception = triggered, so Cassandra try to resolve this version conflict. > But the result is value "Update". >=20 > Here is the suspect root cause, the version conflict resolved based on = time stamp. >=20 > Node C local time is a bit earlier then node A. >=20 > "Update" requests sent from node C with time stamp 00:00:00.050, = "Delete" sent from node A with time stamp 00:00:00.020, which is not = same as the event sequence. >=20 > So the version conflict resolved incorrectly. >=20 > It is true? >=20 > If Yes, then it means, consistency level can secure the conflict been = found, but to solve it correctly, dependence one time synchronization's = accuracy, e.g. NTP ? >=20 >=20 >=20 >=20 --Apple-Mail=_19799891-0A0E-4A7C-9476-4A9185FF137F Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1
Otherwise, it means the = version conflict solving strong depends on global sequence id = (timestamp) which need provide by client ?
Yes. 
If you have an =  area of your data model that has a high degree of concurrency C* = may not be the right match.

In 1.1 we have atomic updates so clients see either the = entire write or none of it. And sometimes you can design a data model = that does mutate shared values, but writes ledger entries instead. See = Matt Denis talk here = http://www.datastax.com/events/cassandrasummit2012/presentations = or this post htt= p://thelastpickle.com/2012/08/18/Sorting-Lists-For-Humans/

Cheers

http://www.thelastpickle.com

On 4/03/2013, at 4:30 PM, Jason Tang <ares.tang@gmail.com> = wrote:

Hi 

The timestamp provided by my client is unix timestamp = (with ntp), and as I said, due to the ntp drift, the local unix = timestamp is not accurately synchronized (compare to my = case).

So for short, client can not = provide global sequence number to indicate the event order.

But I wonder, I configured = Cassandra consistency level as write QUORUM. So for one record, = I suppose Cassandra has the ability to decide the final update = results.

Otherwise, it means the = version conflict solving strong depends on global sequence id = (timestamp) which need provide by client ?


//Tang


2013/3/4 Sylvain Lebresne <sylvain@datastax.com>
The problem is, what is the sequence number you = are talking about is exactly?

Or let me put it = another way: if you do have a sequence number that provides a total = ordering of your operation, then that is exactly what you should use as = your timestamp. What Cassandra calls the timestamp, is exactly what you = call seqID, it's the number Cassandra uses to decide the order of = operation.

Except that in real life, provided you have more = than one client talking to Cassandra, then providing a total ordering of = operation is hard, and in fact not doable efficiently. So in practice, = people use unix timestamp (with ntp) which provide a very good while = cheap approximation of the real life order of operations.

But again, if you do know how to assign a more = precise "timestamp", Cassandra let you use that: you can provid your own = timestamp (using unix timestamp is just the default). The point being, = unix timestamp is the better approximation we have in practice.

--
Sylvain


On Mon, Mar 4, 2013 at 9:26 AM, Jason Tang <ares.tang@gmail.com> wrote:
Hi

  Previous I met a consistency = problem, you can refer the link below for the whole story.

  And after check the code, seems I found = some clue of the problem. Maybe some one can check = this.

  For short, I have Cassandra = cluster (1.0.3), The = consistency level is read/write quorum, replication_factor=  is = 3. 

<= div>  = Here is event sequence:

seqID   = NodeA   NodeB   NodeC
1.     =     New      New       = New
2. =         Update  Update   = Update
3.     =     Delete   Delete    

<= div>When = try to read from NodeB and NodeC, "Digest mismatch" exception = triggered, so Cassandra try to resolve this version = conflict.
But = the result is value "Update".

<= div> Here = is the suspect root cause, the version conflict resolved based = on time stamp.

Node C local time is a bit earlier then node = A.

"Update" requests sent from node C with time = stamp 00:00:00.050, "Delete" sent from node A with time stamp = 00:00:00.020, which is not same as the event sequence.

So the = version conflict resolved incorrectly.

It is = true?

If Yes, then it means, consistency level can = secure the conflict been found, but to solve it correctly, dependence = one time synchronization's accuracy, e.g. NTP ?





= --Apple-Mail=_19799891-0A0E-4A7C-9476-4A9185FF137F--