From user-return-35944-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Wed Aug 14 19:35:28 2013 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4773710EC9 for ; Wed, 14 Aug 2013 19:35:28 +0000 (UTC) Received: (qmail 48069 invoked by uid 500); 14 Aug 2013 19:35:25 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 48021 invoked by uid 500); 14 Aug 2013 19:35:25 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 48009 invoked by uid 99); 14 Aug 2013 19:35:24 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Aug 2013 19:35:24 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of a.e.cobley@dundee.ac.uk designates 65.55.88.12 as permitted sender) Received: from [65.55.88.12] (HELO tx2outboundpool.messaging.microsoft.com) (65.55.88.12) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Aug 2013 19:35:19 +0000 Received: from mail193-tx2-R.bigfish.com (10.9.14.240) by TX2EHSOBE008.bigfish.com (10.9.40.28) with Microsoft SMTP Server id 14.1.225.22; Wed, 14 Aug 2013 19:34:57 +0000 Received: from mail193-tx2 (localhost [127.0.0.1]) by mail193-tx2-R.bigfish.com (Postfix) with ESMTP id 67C7C320092 for ; Wed, 14 Aug 2013 19:34:57 +0000 (UTC) X-Forefront-Antispam-Report: CIP:157.56.249.53;KIP:(null);UIP:(null);IPV:NLI;H:AMSPRD0411HT005.eurprd04.prod.outlook.com;RD:none;EFVD:NLI X-SpamScore: -1 X-BigFish: PS-1(zz98dIc85dhzz1f42h208ch1ee6h1de0h1d18h1fdah2073h1202h1e76h1d1ah1d2ah1fc6hzz1de098h18c673h8275dh1de097hz2fh2a8h668h839hd25hf0ah1288h12a5h12bdh137ah1441h1504h1537h153bh15d0h162dh1631h1758h18e1h1946h19b5h1ad9h1b0ah1bceh1d07h1d0ch1d2eh1d3fh1de9h1dfeh1dffh1e1dh1fe8h1ff5h1155h) Received-SPF: pass (mail193-tx2: domain of dundee.ac.uk designates 157.56.249.53 as permitted sender) client-ip=157.56.249.53; envelope-from=a.e.cobley@dundee.ac.uk; helo=AMSPRD0411HT005.eurprd04.prod.outlook.com ;.outlook.com ; Received: from mail193-tx2 (localhost.localdomain [127.0.0.1]) by mail193-tx2 (MessageSwitch) id 137650889349445_30166; Wed, 14 Aug 2013 19:34:53 +0000 (UTC) Received: from TX2EHSMHS039.bigfish.com (unknown [10.9.14.245]) by mail193-tx2.bigfish.com (Postfix) with ESMTP id 002C0C0041 for ; Wed, 14 Aug 2013 19:34:53 +0000 (UTC) Received: from AMSPRD0411HT005.eurprd04.prod.outlook.com (157.56.249.53) by TX2EHSMHS039.bigfish.com (10.9.99.139) with Microsoft SMTP Server (TLS) id 14.16.227.3; Wed, 14 Aug 2013 19:34:52 +0000 Received: from AMSPRD0411MB437.eurprd04.prod.outlook.com ([169.254.4.20]) by AMSPRD0411HT005.eurprd04.prod.outlook.com ([10.255.62.168]) with mapi id 14.16.0347.000; Wed, 14 Aug 2013 19:34:49 +0000 From: Andrew Cobley To: "user@cassandra.apache.org" Subject: RE: Vnodes, adding a node ? Thread-Topic: Vnodes, adding a node ? Thread-Index: AQHOmSDLr+TiUc/02ESSVXjHwnbjwZmVEfYAgAAFgPA= Date: Wed, 14 Aug 2013 19:34:49 +0000 Message-ID: <820ACB45DF4D1A4D84DFD9388D666CAF57684E34@AMSPRD0411MB437.eurprd04.prod.outlook.com> References: , In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: Content-Type: multipart/alternative; boundary="_000_820ACB45DF4D1A4D84DFD9388D666CAF57684E34AMSPRD0411MB437_" MIME-Version: 1.0 X-OriginatorOrg: dundee.ac.uk X-FOPE-CONNECTOR: Id%0$Dn%*$RO%0$TLS%0$FQDN%$TlsDn% X-Virus-Checked: Checked by ClamAV on apache.org --_000_820ACB45DF4D1A4D84DFD9388D666CAF57684E34AMSPRD0411MB437_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable That looks like the problem. I added the node with that machine as a seed= , realized my mistake and restarted the machine with the correct seed. it j= oined the ring but without streaming. Nodetool rebuild however doesn't see= m to be fixing the situation. I'll remove the node and try re-adding it after cleaning /var/lib/cassandra= . Andy ________________________________ From: Richard Low [richard@wentnet.com] Sent: 14 August 2013 20:11 To: user@cassandra.apache.org Subject: Re: Vnodes, adding a node ? On 14 August 2013 20:02, Andrew Cobley > wrote: I have small test cluster of 2 nodes. I ran a stress test on it and with = nodetool status received the following: /usr/local/bin/apache-cassandra-2.0.0-rc1/log $ ../bin/nodetool status Datacenter: datacenter1 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Status=3DUp/Down |/ State=3DNormal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID = Rack UN 192.168.0.11 141.13 MB 256 49.2% 4d281e2e-efd9-4abf-b= b70-ebdf8e2b4fc3 rack1 UN 192.168.0.10 145.59 MB 256 50.8% 7fc5795a-bd1b-4e42-8= 8d6-024c5216a893 rack1 I then added a third node with no machines writing to the system. Using no= detool status I got the following: /usr/local/bin/apache-cassandra-2.0.0-rc1/log $ ../bin/nodetool status Datacenter: datacenter1 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Status=3DUp/Down |/ State=3DNormal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID = Rack UN 192.168.0.11 141.12 MB 256 32.2% 4d281e2e-efd9-4abf-b= b70-ebdf8e2b4fc3 rack1 UN 192.168.0.10 145.59 MB 256 35.3% 7fc5795a-bd1b-4e42-8= 8d6-024c5216a893 rack1 UN 192.168.0.12 111.9 KB 256 32.5% e5e6d8bd-c652-4c18-8= fa3-3d71471eee65 rack1 Is this correct ? I was under the impression that adding a node to an exis= ting cluster would distribute the load around the cluster. Am I perhaps mis= sing a step or have a config error perhaps ? How did you add the node? It looks like it didn't bootstrap but just joine= d the ring. You need to make sure the node is not set as a seed and that a= uto_bootstrap is true (the default). Alternatively, you could run 'nodetool rebuild' to stream data from the oth= er nodes. Richard. The University of Dundee is a registered Scottish Charity, No: SC015096 --_000_820ACB45DF4D1A4D84DFD9388D666CAF57684E34AMSPRD0411MB437_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
That looks like the problem.  I added the node  with that ma= chine as a seed, realized my mistake and restarted the machine with the cor= rect seed. it joined the ring but without streaming.  Nodetool rebuild however doesn't seem to be fixing the situation.

I'll remove the node and try re-adding it after cleaning /var/lib/cassandra= .

Andy




From: Richard Low [richard@wentnet.com]
Sent: 14 August 2013 20:11
To: user@cassandra.apache.org
Subject: Re: Vnodes, adding a node ?

On 14 August 2013 20:02, Andrew Cobley &= lt;a.e.cobley@= dundee.ac.uk> wrote:
I have  small test cluster of 2 nodes.  I ran a stress test on it= and with nodetool status received the following:

/usr/local/bin/apache-cassandra-2.0.0-rc1/log $ ../bin/nodetool status
Datacenter: datacenter1
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Status=3DUp/Down
|/ State=3DNormal/Leaving/Joining/Moving
--  Address       Load       Tokens &nbs= p;Owns (effective)  Host ID             =                   Rack
UN  192.168.0.11  141.13 MB  256     49.2%   =           4d281e2e-efd9-4abf-bb70-ebdf8e2b4fc3 &nb= sp;rack1
UN  192.168.0.10  145.59 MB  256     50.8%   =           7fc5795a-bd1b-4e42-88d6-024c5216a893 &nb= sp;rack1

I then added a third node with no machines writing to the system.  Usi= ng nodetool status I got the following:

/usr/local/bin/apache-cassandra-2.0.0-rc1/log $ ../bin/nodetool status
Datacenter: datacenter1
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Status=3DUp/Down
|/ State=3DNormal/Leaving/Joining/Moving
--  Address       Load       Tokens &nbs= p;Owns (effective)  Host ID             =                   Rack
UN  192.168.0.11  141.12 MB  256     32.2%   =           4d281e2e-efd9-4abf-bb70-ebdf8e2b4fc3 &nb= sp;rack1
UN  192.168.0.10  145.59 MB  256     35.3%   =           7fc5795a-bd1b-4e42-88d6-024c5216a893 &nb= sp;rack1
UN  192.168.0.12  111.9 KB   256     32.5%   =           e5e6d8bd-c652-4c18-8fa3-3d71471eee65 &nb= sp;rack1

Is this correct ?  I was under the impression that adding a node to an= existing cluster would distribute the load around the cluster. Am I perhap= s missing a step or have a config error perhaps ?

How did you add the node?  It looks like it didn't bootstrap but = just joined the ring.  You need to make sure the node is not set as a = seed and that auto_bootstrap is true (the default).

Alternatively, you could run 'nodetool rebuild' to stream data from th= e other nodes.

Richard. 

The University of Dundee is a registered Sco= ttish Charity, No: SC015096 --_000_820ACB45DF4D1A4D84DFD9388D666CAF57684E34AMSPRD0411MB437_--