Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3A4FE95F9 for ; Sun, 18 Mar 2012 07:05:27 +0000 (UTC) Received: (qmail 14927 invoked by uid 500); 18 Mar 2012 06:47:58 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 14589 invoked by uid 500); 18 Mar 2012 06:47:42 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 14418 invoked by uid 99); 18 Mar 2012 06:47:34 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 18 Mar 2012 06:47:34 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_SOFTFAIL X-Spam-Check-By: apache.org Received-SPF: softfail (athena.apache.org: transitioning domain of caleb@steelhouse.com does not designate 66.46.182.56 as permitted sender) Received: from [66.46.182.56] (HELO relay.ihostexchange.net) (66.46.182.56) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 18 Mar 2012 06:47:27 +0000 Received: from VMBX131.ihostexchange.net ([192.168.40.21]) by hub106.ihostexchange.net ([66.46.182.56]) with mapi; Sun, 18 Mar 2012 02:47:05 -0400 From: Caleb Rackliffe To: "user@cassandra.apache.org" Date: Sun, 18 Mar 2012 02:47:05 -0400 Subject: Token Ring Gaps in a 2 DC Setup Thread-Topic: Token Ring Gaps in a 2 DC Setup Thread-Index: Ac0E0u58F7ib8TG8RwyB0QT0vSbpuA== Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.14.0.111121 acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_CB8AD2F7928Dcalebsteelhousecom_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_CB8AD2F7928Dcalebsteelhousecom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Everyone, I have a cluster using NetworkTopologyStrategy that looks like this: 10.41.116.22 DC1 RAC1 Up Normal 13.21 GB 10= .00% 0 10.54.149.202 DC2 RAC1 Up Normal 6.98 GB = 0.00% 1 10.41.116.20 DC1 RAC2 Up Normal 12.75 GB 10= .00% 17014118300000000000000000000000000000 10.41.116.16 DC1 RAC3 Up Normal 12.62 GB 10= .00% 34028236700000000000000000000000000000 10.54.149.203 DC2 RAC1 Up Normal 6.7 GB = 0.00% 34028236700000000000000000000000000001 10.41.116.18 DC1 RAC4 Up Normal 10.8 GB 1= 0.00% 51042355000000000000000000000000000000 10.41.116.14 DC1 RAC5 Up Normal 10.27 GB 10= .00% 68056473400000000000000000000000000000 10.54.149.204 DC2 RAC1 Up Normal 6.7 GB = 0.00% 68056473400000000000000000000000000001 10.41.116.12 DC1 RAC6 Up Normal 10.58 GB 10= .00% 85070591700000000000000000000000000000 10.41.116.10 DC1 RAC7 Up Normal 10.89 GB 10= .00% 102084710000000000000000000000000000000 10.54.149.205 DC2 RAC1 Up Normal 7.51 GB 0= .00% 102084710000000000000000000000000000001 10.41.116.8 DC1 RAC8 Up Normal 10.48 GB = 10.00% 119098828000000000000000000000000000000 10.41.116.24 DC1 RAC9 Up Normal 10.89 GB 10= .00% 136112947000000000000000000000000000000 10.54.149.206 DC2 RAC1 Up Normal 6.37 GB 0= .00% 136112947000000000000000000000000000001 10.41.116.26 DC1 RAC10 Up Normal 11.17 GB 10.= 00% 153127065000000000000000000000000000000 There are two data centers, one with 10 nodes/2 replicas and one with 5 nod= es/1 replica. What I've attempted to do with my token assignments is have = each node in the smaller DC handle 20% of the keyspace, and this would mean= that I should see roughly equal usage on all 15 boxes. It just doesn't se= em to be happening that way, though. It looks like the "1 replica" nodes a= re carrying about half the data the "2 replica" nodes are. It's almost as = if those nodes are only handling 10% of the keyspace instead of 20%. Does anybody have any suggestions as to what might be going on? I've run n= odetool getendpoints against a bunch of keys, and I always get back three n= odes, so I'm pretty confused. I've also run repair on a few nodes in both = data centers, but the sizes are still vastly different. Thanks! Caleb Rackliffe | Software Developer M 949.981.0159 | caleb@steelhouse.com --_000_CB8AD2F7928Dcalebsteelhousecom_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Hi Everyone,

I have a cluster using NetworkTopologyStrategy that= looks like this:

10.41.116.22   &n= bsp; DC1         RAC1         Up &n= bsp;   Normal  13.21 GB        10.00%  0=                     &nbs= p;                    &nb= sp;
10.54.149.202   DC2         RAC1 &nb= sp;       Up     Normal  6.98 GB    = ;        0.00%   1         &nb= sp;                     &= nbsp;           
10.41.116.20  = ;   DC1         RAC2         U= p     Normal  12.75 GB        10.00% &nb= sp;17014118300000000000000000000000000000      
10= .41.116.16     DC1         RAC3    =     Up     Normal  12.62 GB       =  10.00%  34028236700000000000000000000000000000     &nb= sp;
10.54.149.203   DC2         RAC1 &nb= sp;       Up     Normal  6.7 GB    =          0.00%   340282367000000000000000000= 00000000001      
10.41.116.18     DC1 &= nbsp;       RAC4         Up    = ; Normal  10.8 GB          10.00%  51042= 355000000000000000000000000000000      
10.41.116.= 14     DC1         RAC5       =   Up     Normal  10.27 GB        10= .00%  68056473400000000000000000000000000000      
10.54.149.204   DC2         RAC1   &nbs= p;     Up     Normal  6.7 GB       =       0.00%   68056473400000000000000000000000000001 &n= bsp;    
10.41.116.12     DC1     &= nbsp;   RAC6         Up     Normal  = ;10.58 GB        10.00%  8507059170000000000000000= 0000000000000      
10.41.116.10     DC1=         RAC7         Up   &nb= sp; Normal  10.89 GB        10.00%  102084710= 000000000000000000000000000000     
10.54.149.205 =   DC2         RAC1         Up =     Normal  7.51 GB           0.00%=   102084710000000000000000000000000000001     
10.41.116.8       DC1         RAC8 &n= bsp;        Up     Normal  10.48 GB &nbs= p;      10.00%  119098828000000000000000000000000000000=     
10.41.116.24     DC1    =     RAC9         Up     Normal &nb= sp;10.89 GB        10.00%  13611294700000000000000= 0000000000000000     
10.54.149.206   DC2 &nb= sp;       RAC1         Up     = Normal  6.37 GB           0.00%   136112= 947000000000000000000000000000001     
10.41.116.2= 6     DC1         RAC10       = Up     Normal  11.17 GB        10.00% &n= bsp;153127065000000000000000000000000000000

There are two data centers, one with 10 nodes/2 replicas and one with= 5 nodes/1 replica.  What I've attempted to do with my token assignmen= ts is have each node in the smaller DC handle 20% of the keyspace, and this= would mean that I should see roughly equal usage on all 15 boxes.  It= just doesn't seem to be happening that way, though.  It looks like th= e "1 replica" nodes are carrying about half the data the "2 replica" nodes = are.  It's almost as if those nodes are only handling 10% of the keysp= ace instead of 20%.

Does anybody have any suggesti= ons as to what might be going on?  I've run nodetool getendpoints= against a bunch of keys, and I always get back three nodes, so I'm pretty = confused.  I've also run repair on a few nodes in both data centers, b= ut the sizes are still vastly different.

Thanks!

Cal= eb Rackliffe | Software Developer
M 949= .981.0159 | caleb@steelhouse.com
--_000_CB8AD2F7928Dcalebsteelhousecom_--