Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8725C10362 for ; Wed, 12 Jun 2013 14:32:13 +0000 (UTC) Received: (qmail 70106 invoked by uid 500); 12 Jun 2013 14:32:11 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 69964 invoked by uid 500); 12 Jun 2013 14:32:10 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 69956 invoked by uid 99); 12 Jun 2013 14:32:10 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jun 2013 14:32:10 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of arodrime@gmail.com designates 209.85.215.54 as permitted sender) Received: from [209.85.215.54] (HELO mail-la0-f54.google.com) (209.85.215.54) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jun 2013 14:32:06 +0000 Received: by mail-la0-f54.google.com with SMTP id ec20so7818237lab.41 for ; Wed, 12 Jun 2013 07:31:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=IIvKtCZuXODbbWcgJT2ELJNE2p2nE9STS/QRG35xATc=; b=YPkv7A42SI+ErgTTugHulqLszrE44fUV30XVfOMOlq4k+x8i2l/aEi6PdlwZoLkOu+ v7n2T+EBIKWmcHdDTs9U/GCnJtXfiTdJj9WjTBe67Uw4sd/DWJshhBrd/CfU/walYA77 oJ9a2CBT4K91+OxGg2LwQ1Rxj5cFeVEIjqFJ06sWl3gXL1kJkUpWwtFNzit9s245//lo 9/wivxKEWifFHtFeFVv9Z0DzMeoGVeRKC+BGBoUyQtbEN3cZXCybFQ2jk5Wittyidh7b mYjuOGZQ4K52/luUXmviqAUZ/50y5WXCnhvAa/6P/pgyJaCRx90G5h5XbMLlMfpfi+Jw wEbQ== X-Received: by 10.152.28.66 with SMTP id z2mr3815400lag.5.1371047504607; Wed, 12 Jun 2013 07:31:44 -0700 (PDT) MIME-Version: 1.0 Received: by 10.112.7.168 with HTTP; Wed, 12 Jun 2013 07:31:24 -0700 (PDT) In-Reply-To: References: From: Alain RODRIGUEZ Date: Wed, 12 Jun 2013 16:31:24 +0200 Message-ID: Subject: Re: Multiple data center performance To: user@cassandra.apache.org Cc: comomore@gmail.com Content-Type: multipart/alternative; boundary=089e0158c3d47b7c6304def5df6c X-Virus-Checked: Checked by ClamAV on apache.org --089e0158c3d47b7c6304def5df6c Content-Type: text/plain; charset=ISO-8859-1 "The consistency level does *not* influence which replica are written to. Cassandra always write to all replicas." I am aware of this. My understanding of what Daning said was that client waited for all nodes to acknowledge the write before considering it successful (inducing extra latency) for counters column families while using multiDC, "regardless the consistency level" I might have misunderstood because of Daning's words about CL. Is there something special of this kind regarding counters over multiDC ? Thank you anyway Sylvain 2013/6/12 Sylvain Lebresne > It is the normal behavior, but that's true of any update, not only of > counters. > > The consistency level does *not* influence which replica are written to. > Cassandra always write to all replicas. The consistency level only decides > how replica acknowledgement are waited for. > > -- > Sylvain > > > On Wed, Jun 12, 2013 at 4:56 AM, Alain RODRIGUEZ wrote: > >> "counter will replicate to all replicas during write regardless the >> consistency level" >> >> I that the normal behavior or a bug ? >> >> >> 2013/6/11 Daning Wang >> >>> It is counter caused the problem. counter will replicate to all replicas >>> during write regardless the consistency level. >>> >>> In our case. we don't need to sync the counter across the center. so >>> moving counter to new keyspace and all the replica in one >>> center solved problem. >>> >>> There is option replicate_on_write on table. If you turn that off for >>> counter might have better performance. but you are on high risk to lose >>> data and create inconsistency. I did not try this option. >>> >>> Daning >>> >>> >>> On Sat, Jun 8, 2013 at 6:53 AM, srmore wrote: >>> >>>> I am seeing the similar behavior, in my case I have 2 nodes in each >>>> datacenter and one node always has high latency (equal to the latency >>>> between the two datacenters). When one of the datacenters is shutdown the >>>> latency drops. >>>> >>>> I am curious to know whether anyone else has these issues and if yes >>>> how did to get around it. >>>> >>>> Thanks ! >>>> >>>> >>>> On Fri, Jun 7, 2013 at 11:49 PM, Daning Wang wrote: >>>> >>>>> We have deployed multi-center but got performance issue. When the >>>>> nodes on other center are up, the read response time from clients is 4 or 5 >>>>> times higher. when we take those nodes down, the response time becomes >>>>> normal(compare to the time before we changed to multi-center). >>>>> >>>>> We have high volume on the cluster, the consistency level is one for >>>>> read. so my understanding is most of traffic between data center should be >>>>> read repair. but seems that could not create much delay. >>>>> >>>>> What could cause the problem? how to debug this? >>>>> >>>>> Here is the keyspace, >>>>> >>>>> [default@dsat] describe dsat; >>>>> Keyspace: dsat: >>>>> Replication Strategy: >>>>> org.apache.cassandra.locator.NetworkTopologyStrategy >>>>> Durable Writes: true >>>>> Options: [dc2:1, dc1:3] >>>>> Column Families: >>>>> ColumnFamily: categorization_cache >>>>> >>>>> >>>>> Ring >>>>> >>>>> Datacenter: dc1 >>>>> =============== >>>>> Status=Up/Down >>>>> |/ State=Normal/Leaving/Joining/Moving >>>>> -- Address Load Tokens Owns (effective) Host ID >>>>> Rack >>>>> UN xx.xx.xx..111 59.2 GB 256 37.5% >>>>> 4d6ed8d6-870d-4963-8844-08268607757e rac1 >>>>> DN xx.xx.xx..121 99.63 GB 256 37.5% >>>>> 9d0d56ce-baf6-4440-a233-ad6f1d564602 rac1 >>>>> UN xx.xx.xx..120 66.32 GB 256 37.5% >>>>> 0fd912fb-3187-462b-8c8a-7d223751b649 rac1 >>>>> UN xx.xx.xx..118 63.61 GB 256 37.5% >>>>> 3c6e6862-ab14-4a8c-9593-49631645349d rac1 >>>>> UN xx.xx.xx..117 68.16 GB 256 37.5% >>>>> ee6cdf23-d5e4-4998-a2db-f6c0ce41035a rac1 >>>>> UN xx.xx.xx..116 32.41 GB 256 37.5% >>>>> f783eeef-1c51-4f91-ab7c-a60669816770 rac1 >>>>> UN xx.xx.xx..115 64.24 GB 256 37.5% >>>>> e75105fb-b330-4f40-aa4f-8e6e11838e37 rac1 >>>>> UN xx.xx.xx..112 61.32 GB 256 37.5% >>>>> 2547ee54-88dd-4994-a1ad-d9ba367ed11f rac1 >>>>> Datacenter: dc2 >>>>> =============== >>>>> Status=Up/Down >>>>> |/ State=Normal/Leaving/Joining/Moving >>>>> -- Address Load Tokens Owns (effective) Host ID >>>>> Rack >>>>> DN xx.xx.xx.199 58.39 GB 256 50.0% >>>>> 6954754a-e9df-4b3c-aca7-146b938515d8 rac1 >>>>> DN xx.xx.xx..61 33.79 GB 256 50.0% >>>>> 91b8d510-966a-4f2d-a666-d7edbe986a1c rac1 >>>>> >>>>> >>>>> Thank you in advance, >>>>> >>>>> Daning >>>>> >>>>> >>>> >>> >> > --089e0158c3d47b7c6304def5df6c Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
"The consistency level does *not* influence which replica are w= ritten to. Cassandra always write to all replicas."

I am aware of this.

My understanding of what Daning said was that= client waited for all nodes to acknowledge the write before considering it= successful (inducing extra latency) for=A0counters column families while u= sing multiDC,=A0"regardless the consistency level"=A0 I might have misund= erstood because of Daning's=A0words about CL.

Is there something special of this kind regarding coun= ters over multiDC ?

Thank you anyway Sylvain=


2013/6/12 Sylvain Lebresne <sylvain@datastax.com>
It is the normal behavior, = but that's true of any update, not only of counters.

The consistency level does *not* influence which replica are written to. Ca= ssandra always write to all replicas. The consistency level only decides ho= w replica acknowledgement are waited for.

--
Sylvain
<= div class=3D"h5">


On Wed, Jun 12, 2013 at 4:56 AM, Alain RODRIGUEZ <= arodrime@gmail.com<= /a>> wrote:
"counter will replicate to all r= eplicas during write regardless the consistency level"

I that the normal behavior or a bug ?
=


2013/6/11 Daning Wang <daning@netseer.com>
It is counter caused the pr= oblem. counter will replicate to all replicas during write regardless the c= onsistency level.=A0

In our case. we don't need to sync the counter across th= e center. so moving counter to new keyspace and all the replica in one cent= er=A0solved=A0problem.

There is option=A0replicate_on_write on table. If you t= urn that off for counter might have better performance. but you are on high= risk to lose data and create inconsistency. I did not try this option.

Daning


On Sat, Jun 8, 2013 at 6:5= 3 AM, srmore <comomore@gmail.com> wrote:
I am seeing the s= imilar behavior, in my case I have 2 nodes in each datacenter and one node = always has high latency (equal to the latency between the two datacenters).= When one of the datacenters is shutdown the latency drops.

I am curious to know whether anyone else has these issues and if = yes how did to get around it.

Thanks !


On Fri, Jun 7, 2013 at 11:49 PM, Daning Wang <daning@netseer.com>= wrote:
We have deployed multi-cent= er but got performance issue. When the nodes on other center are up, the re= ad response time from clients is 4 or 5 times higher. when we take those no= des down, the response time becomes normal(compare to the time before we ch= anged to multi-center).

We have high volume on the cluster, the consistency level is= one for read. so my understanding is most of traffic between data center s= hould be read repair. but seems that could not create much delay.

What could cause the problem? how to debug this?
<= div>
Here is the keyspace,

[def= ault@dsat] describe dsat;
Keyspace: dsat:
=A0 Replication Strategy: org.apache.cassandra.lo= cator.NetworkTopologyStrategy
=A0 Durable Writes: true
= =A0 =A0 Options: [dc2:1, dc1:3]
=A0 Column Families:
= =A0 =A0 ColumnFamily: categorization_cache
=A0

Ring

Datace= nter: dc1
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
=
Status=3DUp/Down
|/ State=3DNormal/Leaving/Joining/Moving
-- =A0Address =A0 =A0 =A0 =A0 =A0 Load =A0 =A0 =A0 Tokens =A0Owns (= effective) =A0Host ID =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 Rack
UN =A0xx.xx.xx..111 =A0 =A0 =A0 59.2 GB =A0 =A0256 =A0 =A0 37.5% =A0 = =A0 =A0 =A0 =A0 =A0 4d6ed8d6-870d-4963-8844-08268607757e =A0rac1
= DN =A0xx.xx.xx..121 =A0 =A0 =A0 99.63 GB =A0 256 =A0 =A0 37.5% =A0 =A0 =A0 = =A0 =A0 =A0 9d0d56ce-baf6-4440-a233-ad6f1d564602 =A0rac1
UN =A0xx.xx.xx..120 =A0 =A0 =A0 66.32 GB =A0 256 =A0 =A0 37.5% =A0 =A0= =A0 =A0 =A0 =A0 0fd912fb-3187-462b-8c8a-7d223751b649 =A0rac1
UN = =A0xx.xx.xx..118 =A0 =A0 =A0 63.61 GB =A0 256 =A0 =A0 37.5% =A0 =A0 =A0 =A0= =A0 =A0 3c6e6862-ab14-4a8c-9593-49631645349d =A0rac1
UN =A0xx.xx.xx..117 =A0 =A0 =A0 68.16 GB =A0 256 =A0 =A0 37.5% =A0 =A0= =A0 =A0 =A0 =A0 ee6cdf23-d5e4-4998-a2db-f6c0ce41035a =A0rac1
UN = =A0xx.xx.xx..116 =A0 =A0 =A0 32.41 GB =A0 256 =A0 =A0 37.5% =A0 =A0 =A0 =A0= =A0 =A0 f783eeef-1c51-4f91-ab7c-a60669816770 =A0rac1
UN =A0xx.xx.xx..115 =A0 =A0 =A0 64.24 GB =A0 256 =A0 =A0 37.5% =A0 =A0= =A0 =A0 =A0 =A0 e75105fb-b330-4f40-aa4f-8e6e11838e37 =A0rac1
UN = =A0xx.xx.xx..112 =A0 =A0 =A0 61.32 GB =A0 256 =A0 =A0 37.5% =A0 =A0 =A0 =A0= =A0 =A0 2547ee54-88dd-4994-a1ad-d9ba367ed11f =A0rac1
Datacenter: dc2
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D
Status=3DUp/Down
|/ State=3DNormal/Leaving/Joining/= Moving
-- =A0Address =A0 =A0 =A0 =A0 =A0 Load =A0 =A0 =A0 Tokens = =A0Owns (effective) =A0Host ID =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 Rack
DN =A0xx.xx.xx.199 =A0 =A058.39 GB =A0 256 =A0 =A0 50.0% =A0 =A0 =A0 = =A0 =A0 =A0 6954754a-e9df-4b3c-aca7-146b938515d8 =A0rac1
DN =A0xx= .xx.xx..61 =A0 =A0 =A033.79 GB =A0 256 =A0 =A0 50.0% =A0 =A0 =A0 =A0 =A0 = =A0 91b8d510-966a-4f2d-a666-d7edbe986a1c =A0rac1


Thank you in advance,
Daning






--089e0158c3d47b7c6304def5df6c--