Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 01BF018B13 for ; Sun, 7 Jun 2015 21:31:53 +0000 (UTC) Received: (qmail 80038 invoked by uid 500); 7 Jun 2015 21:31:49 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 80000 invoked by uid 500); 7 Jun 2015 21:31:49 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 79990 invoked by uid 99); 7 Jun 2015 21:31:49 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 07 Jun 2015 21:31:49 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id CEF741A47E5 for ; Sun, 7 Jun 2015 21:31:47 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.901 X-Spam-Level: ** X-Spam-Status: No, score=2.901 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=tink.se Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id G24j8IdZu43v for ; Sun, 7 Jun 2015 21:31:37 +0000 (UTC) Received: from mail-yk0-f180.google.com (mail-yk0-f180.google.com [209.85.160.180]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 8400C20DC7 for ; Sun, 7 Jun 2015 21:31:36 +0000 (UTC) Received: by yked142 with SMTP id d142so44793996yke.3 for ; Sun, 07 Jun 2015 14:30:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tink.se; s=tink; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=p3v3ZJWPvMW1rITQFCt04D03yZpVKPNbNveVF9Q+YeA=; b=Ona1peSJlkqV46ovXF9T8KeZXTfzDUcS1q04dJjTNbPAsPcemGwvv5ngC3zu5eGzDc 4F0jdQ3vYsLD+EXHLqw07P7GVvyF8MCJJcfoQq/AYOm6QXOSno3PtbQM3+rfr/s//y5O /zbgOE8Pg5FSFB3BSR5nNdnJjahSYUA0YskBE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; bh=p3v3ZJWPvMW1rITQFCt04D03yZpVKPNbNveVF9Q+YeA=; b=izMR9kV2ut3UQ9otNe//ce/8/aUHq8kDd+uNSapD/oIA4OKbH1QTV/rzcYWOYMmcmR Fv0E1wD4XPGMYVWaDebJcslQJdNmk6IGvFrYm455yMo3O9ZHuMZB0Fc64pajh/ncvJKq wkMtDIQzf0Kpi/zCzxzdSi/x4+0Qd4929UkNjmc4+mYx/RyXrwZwISlezYymaZs811Jo hWUJQpVgu0OhbY2hgRd4khdjvR9ozApPI0yJqwe9O0NNmcP3ixQSYkyrhUlvrkh7pmIm Do68oqTsY+ETgrS2X4XlmQggknsUzUHPkBPEBN3ComDd8g2gQzNaUj3BYoS4gSzXy1w5 9FdQ== X-Gm-Message-State: ALoCoQmM3KcWSG1PEYjlQ/7qkFXvk7XICRd5xiyzecvqwlqOvIBqy96lrYa42Tu32kvntfVNkRPD X-Received: by 10.129.55.200 with SMTP id e191mr12732250ywa.48.1433712644470; Sun, 07 Jun 2015 14:30:44 -0700 (PDT) MIME-Version: 1.0 Received: by 10.129.27.15 with HTTP; Sun, 7 Jun 2015 14:30:24 -0700 (PDT) In-Reply-To: References: From: Jens Rantil Date: Sun, 7 Jun 2015 23:30:24 +0200 Message-ID: Subject: Re: Newly added node getting more data than expected To: Cassandra Group Content-Type: multipart/alternative; boundary=001a11440148e24ffa0517f43c7b --001a11440148e24ffa0517f43c7b Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi again, I should also point out that `nodetool ring ...` only has one entry for "X.X.X.4" and that that token range is equally large as the other token ranges for the virtual nodes. Let me know if you need any more information from me. Cheers, Jens On Sun, Jun 7, 2015 at 11:19 PM, Jens Rantil wrote: > Hi, > > I had a 3-node (=C3=A0 256 vnodes each) cluster with RF=3D3. I mistakenly= added a > fourth node with "num_tokens: 1" (that is, one vnode). I've always seen > number of vnodes to be proportional to the amount of data a node would > receive. Therefor, I was expecting the node to receive something like > 1/(1+3*256) of the cluster's data. However, this is not the case: > > $ nodetool status mydatacenter > Datacenter: Cassandra > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > Status=3DUp/Down > |/ State=3DNormal/Leaving/Joining/Moving > -- Address Load Tokens Owns (effective) Host ID > Rack > UN X.X.X.2 200.42 GB 256 87.6% > 871968c9-1d6b-4f06-ba90-8b3a8d92dcf0 RAC1 > UN X.X.X.3 198.03 GB 256 53.7% > d7cacd89-8613-4de5-8a5e-a2c53c41ea45 RAC1 > UN X.X.X.4 110.57 GB 1 58.7% > 55daa807-af49-44c5-9742-fe456df621a1 RAC1 > UN X.X.X.5 199.81 GB 256 100.0% > 48cb0782-6c9a-4805-9330-38e192b6b680 RAC1 > > The new node added is "X.X.X.4". Note that I haven't executed `nodetool > cleanup` on the old nodes yet. > > Additional information: > * I am using GossipingPropertyFileSnitch. All nodes are the same > datacenter and rack. > * There are no pending compactions on the node. > > Could anyone explain to me my new node is receiving more data than > expected? Does this have to do with the way the GossipingPropertyFileSnit= ch > decides where to put secondary/tertiary replicas (ie. always "next physic= al > node" in ring)? Do I need to execute `nodetool cleanup` also on newly > commissioned nodes? > > Thanks, > Jens > > -- > Jens Rantil > Backend engineer > Tink AB > > Email: jens.rantil@tink.se > Phone: +46 708 84 18 32 > Web: www.tink.se > > Facebook Linkedin > > Twitter > --=20 Jens Rantil Backend engineer Tink AB Email: jens.rantil@tink.se Phone: +46 708 84 18 32 Web: www.tink.se Facebook Linkedin Twitter --001a11440148e24ffa0517f43c7b Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi again,

I should also point out that = `nodetool ring ...` only has one entry for "X.X.X.4" and that tha= t token range is equally large as the other token ranges for the virtual no= des.

Let me know if you need any more information = from me.

Cheers,
Jens

On Sun, Jun 7, 2015 at 1= 1:19 PM, Jens Rantil <jens.rantil@tink.se> wrote:
Hi,

I had a 3= -node (=C3=A0 256 vnodes each) cluster with RF=3D3. I mistakenly added a fo= urth node with "num_tokens: 1" (that is, one vnode). I've alw= ays seen number of vnodes to be proportional to the amount of data a node w= ould receive. Therefor, I was expecting the node to receive something like = 1/(1+3*256) of the cluster's data. However, this is not the case:
=

$ nodetool status mydatacenter
Datacente= r: Cassandra
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D
Status=3DUp/Down
|/ State=3DNormal/Leaving= /Joining/Moving
-- =C2=A0Address =C2=A0 =C2=A0 Load =C2=A0 =C2=A0= =C2=A0 Tokens =C2=A0Owns (effective) =C2=A0Host ID =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 Rack
UN =C2=A0X.X.X.2 =C2=A0200.42 GB =C2=A0256 =C2=A0= =C2=A0 87.6% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 871968c9-1d6b-4f06-= ba90-8b3a8d92dcf0 =C2=A0RAC1
UN =C2=A0X.X.X.3 =C2=A0198.03 GB =C2= =A0256 =C2=A0 =C2=A0 53.7% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 d7cacd= 89-8613-4de5-8a5e-a2c53c41ea45 =C2=A0RAC1
UN =C2=A0X.X.X.4 =C2=A0= 110.57 GB =C2=A01 =C2=A0 =C2=A0 =C2=A0 58.7% =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 55daa807-af49-44c5-9742-fe456df621a1 =C2=A0RAC1
UN = =C2=A0X.X.X.5 =C2=A0199.81 GB =C2=A0256 =C2=A0 =C2=A0 100.0% =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A048cb0782-6c9a-4805-9330-38e192b6b680 =C2=A0RAC1<= /div>

The new node added is "X.X.X.4". No= te that I haven't executed `nodetool cleanup` on the old nodes yet.

Additional information:
=C2=A0* I am using = GossipingPropertyFileSnitch. All nodes are the same datacenter and rack.
=C2=A0* There are no pending compactions on the node.
Could anyone explain to me my new node is receiving more data t= han expected? Does this have to do with the way the GossipingPropertyFileSn= itch decides where to put secondary/tertiary replicas (ie. always "nex= t physical node" in ring)? Do I need to execute `nodetool cleanup` als= o on newly commissioned nodes?

Thanks,
J= ens

-- <= br>
Jens Rantil
Backend engineer
<= div>Tink AB

Phone: +46 708 84 18 32
Web:=C2=A0www.= tink.se




--
Jens Rantil
Backend en= gineer
Tink AB

Phone: +46 708 84 18 32
Web:=C2= =A0www.tink.se

--001a11440148e24ffa0517f43c7b--