Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 40749 invoked from network); 12 Apr 2011 19:16:20 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 12 Apr 2011 19:16:20 -0000 Received: (qmail 46443 invoked by uid 500); 12 Apr 2011 19:16:18 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 46382 invoked by uid 500); 12 Apr 2011 19:16:18 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 46374 invoked by uid 99); 12 Apr 2011 19:16:18 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Apr 2011 19:16:18 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jonathan.colby@gmail.com designates 209.85.214.44 as permitted sender) Received: from [209.85.214.44] (HELO mail-bw0-f44.google.com) (209.85.214.44) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Apr 2011 19:16:09 +0000 Received: by bwz13 with SMTP id 13so6490792bwz.31 for ; Tue, 12 Apr 2011 12:15:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:subject:mime-version:content-type:from :x-priority:in-reply-to:date:message-id:references:to:x-mailer; bh=vBwCtROjflU+CE1khCWG68FzCuOHVHNSRuZxt5EN6jc=; b=nBaTSFrkPYcLHHbFKhnTM3UrxNL5zanoDEPOhni1LVtHVuElQB8FBWMq9ceAorpBSJ ZAc9HlhYgpjelnYpF32h09cbTnXnH8OZc7zvhuy7VTmQS8548SbIUd/tpPfCDklxJE3y 6gHm7flo4cwddrJtqR/66910lW8i9Ri7NHfZo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:x-priority:in-reply-to:date :message-id:references:to:x-mailer; b=kolehMl8aNw+icjMaYlO6EpXabO45DS+dxza9In/N5QWv1zdcp0I3ViRZM+2qBk5rt BCkmRpVL+8UgphgmoPLZh7cvByiicDjRbvIOVHpxa0eIOsDrSKftPrQhMZb+7DLu6p48 ohERDua8KKWncE2D1jNpjFZqNHvHw9QTfzmhE= Received: by 10.204.19.20 with SMTP id y20mr4722331bka.170.1302635748848; Tue, 12 Apr 2011 12:15:48 -0700 (PDT) Received: from [192.168.1.102] (g226037101.adsl.alicedsl.de [92.226.37.101]) by mx.google.com with ESMTPS id l1sm4036901bkl.13.2011.04.12.12.15.46 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 12 Apr 2011 12:15:47 -0700 (PDT) Subject: Re: Help on decommission Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: multipart/alternative; boundary=Apple-Mail-1-58513855 From: Jonathan Colby X-Priority: 3 In-Reply-To: <5F71895862254D6B9AC22CAB61726E9F@usnynyc1lpt0050> Date: Tue, 12 Apr 2011 21:15:44 +0200 Message-Id: References: <5F71895862254D6B9AC22CAB61726E9F@usnynyc1lpt0050> To: user@cassandra.apache.org X-Mailer: Apple Mail (2.1084) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-1-58513855 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii how long as it been in Leaving status? Is the cluster under stress = test load while you are doing the decommission? On Apr 12, 2011, at 6:53 PM, Baskar Duraikannu wrote: > I have setup a 4 node cluster for testing. When I setup the cluster, I = have setup initial tokens in such a way that each gets 25% of load and = then started the node with autobootstrap=3Dfalse. > =20 > =20 > After all nodes are up, I loaded data using the stress test tool with = replication factor of 3. As per of my testing, I am trying to remove = one of the node using "nodetool decomission" but the node seems to be = stuck in "leaving" status. > =20 > How do I check whether it is doing any work at all? Please help > =20 > =20 > [root@localhost bin]# ./nodetool -h 10.140.22.25 ring > Address Status State Load Owns Token > = 127605887595351923798765477786913079296 > 10.140.22.66 Up Leaving 119.41 MB 25.00% 0 > 10.140.22.42 Up Normal 116.23 MB 25.00% = 42535295865117307932921825928971026432 > 10.140.22.28 Up Normal 119.93 MB 25.00% = 85070591730234615865843651857942052864 > 10.140.22.25 Up Normal 116.21 MB 25.00% = 127605887595351923798765477786913079296 > [root@localhost bin]# ./nodetool -h 10.140.22.66 netstats > Mode: Leaving: streaming data to other nodes > Streaming to: /10.140.22.42 > = /var/lib/cassandra/data/Keyspace1/Standard1-f-1-Data.db/(0,120929157) > progress=3D120929157/120929157 - 100% > /var/lib/cassandra/data/Keyspace1/Standard1-f-2-Data.db/(0,3361291) > progress=3D0/3361291 - 0% > Not receiving any streams. > Pool Name Active Pending Completed > Commands n/a 0 17 > Responses n/a 0 108109 > [root@usnynyc1cass02 bin]# ./nodetool -h 10.140.22.42 netstats > Mode: Normal > Not sending any streams. > Streaming from: /10.140.22.66 > Keyspace1: = /var/lib/cassandra/data/Keyspace1/Standard1-f-2-Data.db/(0,3361291) > progress=3D0/3361291 - 0% > Pool Name Active Pending Completed > Commands n/a 0 11 > Responses n/a 0 107879 > =20 > =20 > Regards, > Baskar --Apple-Mail-1-58513855 Content-Transfer-Encoding: 7bit Content-Type: text/html; charset=us-ascii how long as it been in Leaving status?   Is the cluster under stress test load while you are doing the decommission?

On Apr 12, 2011, at 6:53 PM, Baskar Duraikannu wrote:

I have setup a 4 node cluster for testing. When I setup the cluster, I have setup initial tokens in such a way that each gets 25% of load and then started the node with autobootstrap=false.
 
 
After all nodes are up, I loaded data using the stress test tool with replication factor of 3.  As per of my testing, I am trying to remove one of the node using "nodetool decomission" but the node seems to be stuck in "leaving" status.
 
How do I check whether it is doing any work at all? Please help
 
 
[root@localhost bin]# ./nodetool -h 10.140.22.25 ring
Address         Status State   Load            Owns    Token
                                                       127605887595351923798765477786913079296
10.140.22.66    Up     Leaving 119.41 MB       25.00%  0
10.140.22.42    Up     Normal  116.23 MB       25.00%  42535295865117307932921825928971026432
10.140.22.28    Up     Normal  119.93 MB       25.00%  85070591730234615865843651857942052864
10.140.22.25    Up     Normal  116.21 MB       25.00%  127605887595351923798765477786913079296
[root@localhost bin]# ./nodetool -h 10.140.22.66 netstats
Mode: Leaving: streaming data to other nodes
Streaming to: /10.140.22.42
   /var/lib/cassandra/data/Keyspace1/Standard1-f-1-Data.db/(0,120929157)
         progress=120929157/120929157 - 100%
   /var/lib/cassandra/data/Keyspace1/Standard1-f-2-Data.db/(0,3361291)
         progress=0/3361291 - 0%
Not receiving any streams.
Pool Name                    Active   Pending      Completed
Commands                        n/a         0             17
Responses                       n/a         0         108109
[root@usnynyc1cass02 bin]# ./nodetool -h 10.140.22.42 netstats
Mode: Normal
Not sending any streams.
Streaming from: /10.140.22.66
   Keyspace1: /var/lib/cassandra/data/Keyspace1/Standard1-f-2-Data.db/(0,3361291)
         progress=0/3361291 - 0%
Pool Name                    Active   Pending      Completed
Commands                        n/a         0             11
Responses                       n/a         0         107879
 
 
Regards,
Baskar

--Apple-Mail-1-58513855--