Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5CBD410090 for ; Fri, 1 Nov 2013 17:55:04 +0000 (UTC) Received: (qmail 67291 invoked by uid 500); 1 Nov 2013 17:55:01 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 67228 invoked by uid 500); 1 Nov 2013 17:55:01 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 67220 invoked by uid 99); 1 Nov 2013 17:55:00 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Nov 2013 17:55:00 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of rakeshxp@gmail.com designates 209.85.215.49 as permitted sender) Received: from [209.85.215.49] (HELO mail-la0-f49.google.com) (209.85.215.49) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Nov 2013 17:54:54 +0000 Received: by mail-la0-f49.google.com with SMTP id ev20so1900806lab.22 for ; Fri, 01 Nov 2013 10:54:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=q1FGiJOHUznGojwvy+YseVkbcPbSK47YM5yx8Cvt3wA=; b=rklaCHWABw9mxT19LFZJfcG7efcIcY+lxArcsYx5naS+0xHxHdc+v7Opsd1cjbQ1Ru 1N5IHq8JIjmm5heBhAlbfog8LX31BkAIcf9wMdCbU7YS7uSA5yQsoO9ArD0ZUpNfUUo+ 3xHItfbdcrR3UDu/z5oJ1fuqSnc/FQux3QfxftdXQZA51aSQsImHgVshsAQA0xQ3MPti lFWRU+vJuyz4WUUJKvbWl3Tk+gPawVc3nvegYs0gZViXA8r0B5YmIvuyHXGgfgQik6x5 4iqnO4uM3SUnEgE+0/igPgNamE3cLGJht4ojgrMGXGfegqKv04JVhPPU15f2+K+/yZ/W qF7A== X-Received: by 10.152.180.139 with SMTP id do11mr2697084lac.23.1383328474115; Fri, 01 Nov 2013 10:54:34 -0700 (PDT) MIME-Version: 1.0 Received: by 10.152.5.34 with HTTP; Fri, 1 Nov 2013 10:54:14 -0700 (PDT) In-Reply-To: References: From: Rakesh Rajan Date: Fri, 1 Nov 2013 23:24:14 +0530 Message-ID: Subject: Re: High loads only on one node in the cluster To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a1135eb6c4eb43704ea214269 X-Virus-Checked: Checked by ClamAV on apache.org --001a1135eb6c4eb43704ea214269 Content-Type: text/plain; charset=ISO-8859-1 Tyler, Thanks for the explanation. The objective is not to have a perfectly balanced US-East and SG DC clusters. SG DC cluster is just a backup cluster and hence has lesser nodes than US-East cluster. What we are trying to figure out is the imbalance between the 6 nodes within US-East itself. I'll try to correct the 6 nodes with US-East to proper racks and check. In addition, as I mentioned earlier, do you see any issues with the dynamic snitch attribute score? I see that node has high score, but what value of dynamic_snitch_badness_threshold should I set so that other replicas can get the traffic? ( that node has >50% higher score than all other nodes ) On Fri, Nov 1, 2013 at 10:04 PM, Tyler Hobbs wrote: > > On Fri, Nov 1, 2013 at 5:07 AM, Rakesh Rajan wrote: > >> >> 1) By alternating racks, do you mean to alternate racks between all nodes >> in a single DC v/s multiple DCs? AWS EastCoast has 4 AZs >> and Singapore has 2 AZs. So is the final solution something like this: >> ip11 - East Coast - m1.xlarge / us-east-1b - Token: 0 >> ip21 - Singapore - m1.xlarge / ap-southeast-1a - Token: 1001 >> ip12 - East Coast - m1.xlarge / us-east-*1c* - >> Token: 28356863910078205288614550619314017621 >> ip13 - East Coast - m1.xlarge / us-east-*1d* - >> Token: 56713727820156410577229101238628035241 >> ip22 - Singapore - m1.xlarge / ap-southeast-1b - >> Token: 56713727820156410577229101238628036241 >> ip14 - East Coast - m1.xlarge / us-east-*1a* - >> Token: 85070591730234615865843651857942052863 >> ip15 - East Coast - m1.xlarge / us-east-*1b* - >> Token: 113427455640312821154458202477256070484 >> ip23 - Singapore - m1.xlarge / ap-southeast-*1a* - >> Token: 113427455640312821154458202477256071484 >> ip16 - East Coast - m1.xlarge / us-east-*1c* - >> Token: 141784319550391026443072753096570088105 >> >> Is this what you had suggested? >> > > That would be more balanced than your current setup, but it would still be > unbalanced, especially the ap-southeast DC. To have a perfectly balanced > cluster with multiple racks, you need to a) have the same number of nodes > on each rack, and b) alternate racks within each DC. Your new layout would > meet requirement (b), but not (a). This is why I suggest using the same > rack for all nodes. > > > -- > Tyler Hobbs > DataStax > --001a1135eb6c4eb43704ea214269 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Tyler,

Thanks for the explanation. The = objective is not to have a perfectly balanced US-East and SG DC clusters. S= G DC cluster is just a backup cluster and hence has lesser nodes than US-Ea= st cluster. What we are trying to figure out is the imbalance between the 6= nodes within US-East itself. I'll try to correct the 6 nodes with US-E= ast to proper racks and check.

In addition, as I mentioned earlier, do you see a= ny issues with the dynamic snitch=A0attribute score? I see that node has high score, but w= hat value of=A0dynamic_snitch_badness_threshold should I set so that other replicas can get= the traffic? ( that node has >50% higher score than all other nodes )


On Fri,= Nov 1, 2013 at 10:04 PM, Tyler Hobbs <tyler@datastax.com> = wrote:
=

On Fri, Nov 1, 2013 at 5:0= 7 AM, Rakesh Rajan <rakeshxp@gmail.com> wrote:

1) By alternating racks, do you mean to alternate racks between all nodes = in a single DC v/s multiple DCs? AWS EastCoast has 4 AZs and=A0Singapore=A0= has=A02 AZs. So is the final solution something like this:=A0
ip11 = - East Coast -=A0m1.xlarge / us-eas= t-1b =A0 =A0 =A0 =A0 - Token: 0
ip21 - Singapore =A0-=A0m1.xlarg= e / ap-southeast-1a=A0- Token: 1001=
ip12 - Eas= t Coast -=A0m1.xlarge / us-east-= 1c =A0 =A0 =A0 =A0 - Token:=A028356863910078205288614550619314017621
ip13 - Eas= t Coast -=A0m1.xlarge / us-east-= 1d =A0 =A0 =A0 =A0 - Token:=A056713727820156410577229101238628035241
ip22 -=A0<= /span>Singapore= =A0=A0-= =A0m1.xlarge / ap-south= east-1b=A0- Token:=A056713727820156= 410577229101238628036241
ip14 - East Coast -=A0m1.xlarge = / us-east-1a =A0 =A0 =A0 =A0 - Token:=A08507059173023461586584365185= 7942052863
ip15 - East Coast -=A0m1.xlarge = / us-east-1b =A0 =A0 =A0 =A0 - Token:=A01134274556403128211544582024= 77256070484
ip23 -=A0Singapore=A0 -=A0m1.xlarge /=A0<= span style=3D"font-family:arial,sans-serif">ap-southeast-1a=A0- Token:=A0113427455640312821154458202477= 256071484
ip16 - East Coast -=A0m1.xlarge = / us-east-1c =A0 =A0 =A0 =A0 - Token:=A01417843195503910264430727530= 96570088105

Is this what you had suggested?=A0

That would be more balanced than your current setu= p, but it would still be unbalanced, especially the ap-southeast DC.=A0 To = have a perfectly balanced cluster with multiple racks, you need to a) have = the same number of nodes on each rack, and b) alternate racks within each D= C.=A0 Your new layout would meet requirement (b), but not (a).=A0 This is w= hy I suggest using the same rack for all nodes.


--
Tyler Hobbs
DataStax

--001a1135eb6c4eb43704ea214269--