Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B46A7109F4 for ; Mon, 7 Apr 2014 13:53:22 +0000 (UTC) Received: (qmail 53020 invoked by uid 500); 7 Apr 2014 13:53:18 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 52781 invoked by uid 500); 7 Apr 2014 13:53:13 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 52773 invoked by uid 99); 7 Apr 2014 13:53:12 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Apr 2014 13:53:12 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,HTML_OBFUSCATE_05_10,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy includes SPF record at spf.trusted-forwarder.org) Received: from [209.85.223.171] (HELO mail-ie0-f171.google.com) (209.85.223.171) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Apr 2014 13:53:07 +0000 Received: by mail-ie0-f171.google.com with SMTP id ar20so6237241iec.2 for ; Mon, 07 Apr 2014 06:52:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=3nR8sDdJDTQLSvTWZ5SblW7yjQTplpd+hIhsq+twwIc=; b=WnLn2BOTmHEyKsgGuQQX0U6Tu8+6ADvxe0l0WqM5s0ht33mzMJ5yeMXi2p9iVPBiLV Xk+fY2PjVtlAegOhpYPwJ8h/fVHlvhLlT17pxw4+8Nkppt0Ap/rZ43UlL7Sguq50hXM7 dGrEK2ahTmMIMdZswHOrfLSJxpvANULW6y0BfrafC9EaXsHFAME/XqTzL17b4xsBGvE1 22EnADb7fvT5ShgK2pzygSO/uUJ5yVc1T/xqk7M/ErTx/vmM031SL9F+ZcMkFG9t2Y0O 526haxtnxa6lBskMjM6o0ioHhb0D0pOcb+jzPT7p9Js6xf2XeG+v5xBhc4pLoJmRU1T/ LxQg== X-Gm-Message-State: ALoCoQkMg8VY2OR0UoH9SW4WZFfs8DHQK4V3UHxa0+68Hnkg9TDwjW7e02rxi603f6FlnnaLvo9n MIME-Version: 1.0 X-Received: by 10.50.109.230 with SMTP id hv6mr20526536igb.9.1396878766298; Mon, 07 Apr 2014 06:52:46 -0700 (PDT) Received: by 10.64.35.33 with HTTP; Mon, 7 Apr 2014 06:52:46 -0700 (PDT) X-Originating-IP: [207.237.102.71] Received: by 10.64.35.33 with HTTP; Mon, 7 Apr 2014 06:52:46 -0700 (PDT) In-Reply-To: References: Date: Mon, 7 Apr 2014 09:52:46 -0400 Message-ID: Subject: Re: Why is my cluster imbalanced ? From: Tupshin Harper To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=089e013a1d8ea8f59b04f6742e44 X-Virus-Checked: Checked by ClamAV on apache.org --089e013a1d8ea8f59b04f6742e44 Content-Type: text/plain; charset=ISO-8859-1 I recommend rf=3 for most situations, and it would certainly be appropriate here. Just remember to add a third rack, and maintain the able number of nodes in each rack. -Tupshin On Apr 7, 2014 9:49 AM, "Oleg Dulin" wrote: > Tupshin: > > For EC2, 3 us-east, would you recommend RF=3 ? That would make sense, > wouldn't it... > > That's what I'll do for production. > > Oleg > > On 2014-04-07 12:23:51 +0000, Tupshin Harper said: > > Your us-east datacenter, has RF=2, and 2 racks, which is the right way >> to do it (I would rarely recommend using a different number of racks >> than your RF). But by having three nodes on one rack (1b) and only one >> on the other(1a), you are telling Cassandra to distribute the data so >> that no two copies of the same partition exist on the same rack. >> >> So with rack ownership of 100% and 100% respectively, there is no even >> way to distribute your data among those four nodes. >> >> tl;dr Switch node 2 to rack 1a. >> >> -Tupshin >> >> >> >> On Mon, Apr 7, 2014 at 8:08 AM, Oleg Dulin wrote: >> >>> I added two more nodes on Friday, and moved tokens around. >>> >>> For four nodes, the tokesn should be: >>> >>> Node #1: 0 >>> Node #2: 42535295865117307932921825928971026432 >>> Node #3: 85070591730234615865843651857942052864 >>> Node #4: 127605887595351923798765477786913079296 >>> >>> And yet my ring status shows this (for a specific keyspace). RF=2. >>> >>> Datacenter: us-east >>> ========== >>> Replicas: 2 >>> >>> Address Rack Status State Load Owns >>> Token >>> >>> 42535295865117307932921825928971026432 >>> x.x.x.1 1b Up Normal 13.51 GB 25.00% >>> 127605887595351923798765477786913079296 >>> x.x.x.2 1b Up Normal 4.46 GB 25.00% >>> 85070591730234615865843651857942052164 >>> x.x.x.3 1a Up Normal 62.58 GB 100.00% 0 >>> x.x.x.4 1b Up Normal 66.71 GB 50.00% >>> 42535295865117307932921825928971026432 >>> >>> Datacenter: us-west >>> ========== >>> Replicas: 1 >>> >>> Address Rack Status State Load Owns >>> Token >>> >>> x.x.x.5 1b Up Normal 62.72 GB 100.00% >>> 100 >>> -- >>> Regards, >>> Oleg Dulin >>> http://www.olegdulin.com >>> >> > > -- > Regards, > Oleg Dulin > http://www.olegdulin.com > > > --089e013a1d8ea8f59b04f6742e44 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

I recommend rf=3D3 for most situations, and it would certain= ly be appropriate here.

Just remember to add a third rack, and maintain the able num= ber of nodes in each rack.

-Tupshin

On Apr 7, 2014 9:49 AM, "Oleg Dulin" &= lt;oleg.dulin@gmail.com> wro= te:
Tupshin:

For EC2, 3 us-east, would you recommend RF=3D3 ? That would make sense, wou= ldn't it...

That's what I'll do for production.

Oleg

On 2014-04-07 12:23:51 +0000, Tupshin Harper said:

Your us-east datacenter, has RF=3D2, and 2 racks, which is the right way to do it (I would rarely recommend using a different number of racks
than your RF). But by having three nodes on one rack (1b) and only one
on the other(1a), you are telling Cassandra to distribute the data so
that no two copies of the same partition exist on the same rack.

So with rack ownership of 100% and 100% respectively, there is no even
way to distribute your data among those four nodes.

tl;dr Switch node 2 to rack 1a.

-Tupshin



On Mon, Apr 7, 2014 at 8:08 AM, Oleg Dulin <oleg.dulin@gmail.com> wrote:
I added two more nodes on Friday, and moved tokens around.

For four nodes, the tokesn should be:

Node #1: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A00
Node #2: =A0 42535295865117307932921825928971026432
Node #3: =A0 85070591730234615865843651857942052864
Node #4: =A0127605887595351923798765477786913079296

And yet my ring status shows this (for a specific keyspace). RF=3D2.

Datacenter: us-east
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Replicas: 2

Address =A0 =A0 =A0 =A0Rack =A0 =A0 =A0 =A0Status State =A0 Load =A0 =A0 = =A0 =A0 =A0 =A0Owns
Token

42535295865117307932921825928971026432
x.x.x.1 =A01b =A0 =A0 =A0 =A0 =A0Up =A0 =A0 Normal =A013.51 GB =A0 =A0 =A0 = =A025.00%
127605887595351923798765477786913079296
x.x.x.2 =A01b =A0 =A0 =A0 =A0 =A0Up =A0 =A0 Normal =A04.46 GB =A0 =A0 =A0 = =A0 25.00%
85070591730234615865843651857942052164
x.x.x.3 =A01a =A0 =A0 =A0 =A0 =A0Up =A0 =A0 Normal =A062.58 GB =A0 =A0 =A0 = =A0100.00% =A0 =A0 =A0 =A0 =A0 =A0 0
x.x.x.4 =A01b =A0 =A0 =A0 =A0 =A0Up =A0 =A0 Normal =A066.71 GB =A0 =A0 =A0 = =A050.00%
42535295865117307932921825928971026432

Datacenter: us-west
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Replicas: 1

Address =A0 =A0 =A0 =A0Rack =A0 =A0 =A0 =A0Status State =A0 Load =A0 =A0 = =A0 =A0 =A0 =A0Owns
Token

x.x.x.5 =A0 1b =A0 =A0 =A0 =A0 =A0Up =A0 =A0 Normal =A062.72 GB =A0 =A0 =A0= =A0100.00% =A0 =A0 =A0 =A0 =A0 =A0 100
--
Regards,
Oleg Dulin
http://www.olegdulin= .com


--
Regards,
Oleg Dulin
http://www.olegdulin= .com


--089e013a1d8ea8f59b04f6742e44--