Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CC83F7570 for ; Tue, 23 Aug 2011 08:46:13 +0000 (UTC) Received: (qmail 746 invoked by uid 500); 23 Aug 2011 08:46:10 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 99522 invoked by uid 500); 23 Aug 2011 08:45:57 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 98850 invoked by uid 99); 23 Aug 2011 08:45:55 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Aug 2011 08:45:55 +0000 X-ASF-Spam-Status: No, hits=3.3 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,TRACKER_ID X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a58.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Aug 2011 08:45:48 +0000 Received: from homiemail-a58.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a58.g.dreamhost.com (Postfix) with ESMTP id E22677D805B for ; Tue, 23 Aug 2011 01:45:27 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=ipGSMaq79M dKhkJ/AsWpebSjHzqofNGKOwUmEBw7kzRYItVXaSve9j8obGaXU/YXE2OJldbJBX D1ys/+Q5BRXcIYEwYL0bUTXiEwvwno9SA4xhYB3guTjLSijWAL2aZ/j8rXIZ6UdQ T9NQ3pST7/CRQfNanIJIs7pZqgmfrrpMs= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=B+UDsCRE9bOxtw3u 6hWey78doGM=; b=eAErCg/jWWGMQEetuALQcQpOICKPowxYgwFgcNiqYPAC+mI1 KZ7Daj4lnvBtiYFUl+SyR48zYFpvPugFAgxIPbKV7uAXuDuxuPRPk6sah/lafAtR 0s8hQ0IYwC6158D78zRdtYo8TRjdcniEnnzanLL+iRduMCtU6BL/ER6KiEQ= Received: from [172.16.1.4] (222-152-100-67.jetstream.xtra.co.nz [222.152.100.67]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a58.g.dreamhost.com (Postfix) with ESMTPSA id 62CD77D8058 for ; Tue, 23 Aug 2011 01:45:26 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1244.3) Content-Type: multipart/alternative; boundary="Apple-Mail=_B53BA189-4CA0-4543-B5B2-6B6E59797830" Subject: Re: Completely removing a node from the cluster Date: Tue, 23 Aug 2011 20:45:23 +1200 In-Reply-To: <076926A9-B9E1-4CC3-B858-C116C97BDE09@gmail.com> To: user@cassandra.apache.org References: <376CEC01195C894CB9F8A3C274029A96AF25338F@FISH-EX2K10-01.azaleos.net> <593A1215-C630-4D6B-B905-4779389A782B@thelastpickle.com> <376CEC01195C894CB9F8A3C274029A96AF256B8B@FISH-EX2K10-01.azaleos.net> <504F4C34-7C5C-43D5-8821-18758D389F16@thelastpickle.com> <376CEC01195C894CB9F8A3C274029A96AF256DAD@FISH-EX2K10-01.azaleos.net> <376CEC01195C894CB9F8A3C274029A96AF258687@FISH-EX2K10-01.azaleos.net> <81FAAD69-6DA8-41A9-86E0-F5B66D55FD34@thelastpickle.com> <076926A9-B9E1-4CC3-B858-C116C97BDE09@gmail.com> Message-Id: <31E4C10E-C4CD-4CE1-A1D4-61FD04FDD4CF@thelastpickle.com> X-Mailer: Apple Mail (2.1244.3) --Apple-Mail=_B53BA189-4CA0-4543-B5B2-6B6E59797830 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii I normally link to the data stax article to avoid having to actually = write those words :) = http://www.datastax.com/docs/0.8/troubleshooting/index#view-of-ring-differ= s-between-some-nodes A ----------------- Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 23/08/2011, at 7:45 PM, Jonathan Colby wrote: > I ran into this. I also tried log_ring_state=3Dfalse which also did = not help. The way I got through this was to stop the entire cluster = and start the nodes one-by-one. =20 >=20 > I realize this is not a practical solution for everyone, but if you = can afford to stop the cluster for a few minutes, it's worth a try. >=20 >=20 > On Aug 23, 2011, at 9:26 AM, aaron morton wrote: >=20 >> I'm running low on ideas for this one. Anyone else ?=20 >>=20 >> If the phantom node is not listed in the ring, other nodes should not = be storing hints for it. You can see what nodes they are storing hints = for via JConsole.=20 >>=20 >> You can try a rolling restart passing the JVM opt = -Dcassandra.load_ring_state=3Dfalse However if the phantom node is been = passed around in the gossip state it will probably just come back again.=20= >>=20 >> Cheers >>=20 >>=20 >> ----------------- >> Aaron Morton >> Freelance Cassandra Developer >> @aaronmorton >> http://www.thelastpickle.com >>=20 >> On 23/08/2011, at 3:49 PM, Bryce Godfrey wrote: >>=20 >>> Could this ghost node be causing my hints column family to grow to = this size? I also crash after about 24 hours due to commit logs growth = taking up all the drive space. A manual nodetool flush keeps it under = control though. >>>=20 >>>=20 >>> Column Family: HintsColumnFamily >>> SSTable count: 6 >>> Space used (live): 666480352 >>> Space used (total): 666480352 >>> Number of Keys (estimate): 768 >>> Memtable Columns Count: 1043 >>> Memtable Data Size: 461773 >>> Memtable Switch Count: 3 >>> Read Count: 38 >>> Read Latency: 131.289 ms. >>> Write Count: 582108 >>> Write Latency: 0.019 ms. >>> Pending Tasks: 0 >>> Key cache capacity: 7 >>> Key cache size: 6 >>> Key cache hit rate: 0.8333333333333334 >>> Row cache: disabled >>> Compacted row minimum size: 2816160 >>> Compacted row maximum size: 386857368 >>> Compacted row mean size: 120432714 >>>=20 >>> Is there a way for me to manually remove this dead node? >>>=20 >>> -----Original Message----- >>> From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]=20 >>> Sent: Sunday, August 21, 2011 9:09 PM >>> To: user@cassandra.apache.org >>> Subject: RE: Completely removing a node from the cluster >>>=20 >>> It's been at least 4 days now. >>>=20 >>> -----Original Message----- >>> From: aaron morton [mailto:aaron@thelastpickle.com]=20 >>> Sent: Sunday, August 21, 2011 3:16 PM >>> To: user@cassandra.apache.org >>> Subject: Re: Completely removing a node from the cluster >>>=20 >>> I see the mistake I made about ring, gets the endpoint list from the = same place but uses the token's to drive the whole process.=20 >>>=20 >>> I'm guessing here, don't have time to check all the code. But there = is a 3 day timeout in the gossip system. Not sure if it applies in this = case.=20 >>>=20 >>> Anyone know ? >>>=20 >>> Cheers >>>=20 >>> ----------------- >>> Aaron Morton >>> Freelance Cassandra Developer >>> @aaronmorton >>> http://www.thelastpickle.com >>>=20 >>> On 22/08/2011, at 6:23 AM, Bryce Godfrey wrote: >>>=20 >>>> Both .2 and .3 list the same from the mbean that Unreachable is = empty collection, and Live node lists all 3 nodes still: >>>> 192.168.20.2 >>>> 192.168.20.3 >>>> 192.168.20.1 >>>>=20 >>>> The removetoken was done a few days ago, and I believe the remove = was done from .2 >>>>=20 >>>> Here is what ring outlook looks like, not sure why I get that token = on the empty first line either: >>>> Address DC Rack Status State Load = Owns Token >>>> = 85070591730234615865843651857942052864 >>>> 192.168.20.2 datacenter1 rack1 Up Normal 79.53 GB = 50.00% 0 >>>> 192.168.20.3 datacenter1 rack1 Up Normal 42.63 GB = 50.00% 85070591730234615865843651857942052864 >>>>=20 >>>> Yes, both nodes show the same thing when doing a describe cluster, = that .1 is unreachable. >>>>=20 >>>>=20 >>>> -----Original Message----- >>>> From: aaron morton [mailto:aaron@thelastpickle.com]=20 >>>> Sent: Sunday, August 21, 2011 4:23 AM >>>> To: user@cassandra.apache.org >>>> Subject: Re: Completely removing a node from the cluster >>>>=20 >>>> Unreachable nodes in either did not respond to the message or were = known to be down and were not sent a message.=20 >>>> The way the node lists are obtained for the ring command and = describe cluster are the same. So it's a bit odd.=20 >>>>=20 >>>> Can you connect to JMX and have a look at the = o.a.c.db.StorageService MBean ? What do the LiveNode and UnrechableNodes = attributes say ?=20 >>>>=20 >>>> Also how long ago did you remove the token and on which machine? Do = both 20.2 and 20.3 think 20.1 is still around ?=20 >>>>=20 >>>> Cheers >>>>=20 >>>>=20 >>>> ----------------- >>>> Aaron Morton >>>> Freelance Cassandra Developer >>>> @aaronmorton >>>> http://www.thelastpickle.com >>>>=20 >>>> On 20/08/2011, at 9:48 AM, Bryce Godfrey wrote: >>>>=20 >>>>> I'm on 0.8.4 >>>>>=20 >>>>> I have removed a dead node from the cluster using nodetool = removetoken command, and moved one of the remaining nodes to rebalance = the tokens. Everything looks fine when I run nodetool ring now, as it = only lists the remaining 2 nodes and they both look fine, owning 50% of = the tokens. >>>>>=20 >>>>> However, I can still see it being considered as part of the = cluster from the Cassandra-cli (192.168.20.1 being the removed node) and = I'm worried that the cluster is still queuing up hints for the node, or = any other issues it may cause: >>>>>=20 >>>>> Cluster Information: >>>>> Snitch: org.apache.cassandra.locator.SimpleSnitch >>>>> Partitioner: org.apache.cassandra.dht.RandomPartitioner >>>>> Schema versions: >>>>> dcc8f680-caa4-11e0-0000-553d4dced3ff: [192.168.20.2, = 192.168.20.3] >>>>> UNREACHABLE: [192.168.20.1] >>>>>=20 >>>>>=20 >>>>> Do I need to do something else to completely remove this node? >>>>>=20 >>>>> Thanks, >>>>> Bryce >>>>=20 >>>=20 >>=20 >=20 --Apple-Mail=_B53BA189-4CA0-4543-B5B2-6B6E59797830 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii I = normally link to the data stax article to avoid having to actually write = those words :)

A
http://www.thelastpickle.com

On 23/08/2011, at 7:45 PM, Jonathan Colby wrote:

I ran = into this.  I also tried log_ring_state=3Dfalse which also did not = help.   The way I got through this was to stop the entire = cluster and start the nodes one-by-one.   

I realize = this is not a practical solution for everyone, but if you can afford to = stop the cluster for a few minutes, it's worth a try.


On Aug = 23, 2011, at 9:26 AM, aaron morton wrote:

I'm running low on ideas for this one. Anyone else ? =

If the phantom node is not listed in the ring, other nodes = should not be storing hints for it. You can see what nodes they are = storing hints for via JConsole.

You can try a = rolling restart passing the JVM opt -Dcassandra.load_ring_state=3Dfalse = However if the phantom node is been passed around in the gossip state it = will probably just come back again.

Cheers


-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
=

On 23/08/2011, at 3:49 PM, Bryce Godfrey = wrote:

Could this ghost node be causing my hints column family to = grow to this size?  I also crash after about 24 hours due to commit = logs growth taking up all the drive space.  A manual nodetool flush = keeps it under control though.


=             &n= bsp;Column Family: = HintsColumnFamily
=             &n= bsp;SSTable count: 6
=             &n= bsp;Space used (live): = 666480352
=             &n= bsp;Space used (total): = 666480352
=             &n= bsp;Number of Keys (estimate): = 768
=             &n= bsp;Memtable Columns Count: = 1043
=             &n= bsp;Memtable Data Size: 461773
=             &n= bsp;Memtable Switch Count: 3
=             &n= bsp;Read Count: 38
=             &n= bsp;Read Latency: 131.289 ms.
=             &n= bsp;Write Count: 582108
=             &n= bsp;Write Latency: 0.019 ms.
=             &n= bsp;Pending Tasks: 0
=             &n= bsp;Key cache capacity: 7
=             &n= bsp;Key cache size: 6
=             &n= bsp;Key cache hit rate: = 0.8333333333333334
=             &n= bsp;Row cache: disabled
=             &n= bsp;Compacted row minimum size: = 2816160
=             &n= bsp;Compacted row maximum size: = 386857368
=             &n= bsp;Compacted row mean size: = 120432714

Is there a way for me to = manually remove this dead node?

-----Original = Message-----
From: Bryce Godfrey = [mailto:Bryce.Godfrey@azaleos.com] =
Sent: Sunday, August 21, 2011 9:09 = PM
To: user@cassandra.apache.org
Subject: RE: Completely removing a node from the = cluster

It's been at least 4 days = now.

-----Original = Message-----
From: aaron morton = [mailto:aaron@thelastpickle.com] =
Sent: Sunday, August 21, 2011 3:16 = PM
To: user@cassandra.apache.org
Subject: Re: Completely removing a node from the = cluster

I see the mistake I made about = ring, gets the endpoint list from the same place but uses the token's to = drive the whole process.

I'm guessing here, don't have = time to check all the code. But there is a 3 day timeout in the gossip = system. Not sure if it applies in this case. =

Anyone know = ?

Cheers

-----------------
Aaron = Morton
Freelance Cassandra = Developer
@aaronmorton
http://www.thelastpickle.com
=

On 22/08/2011, at 6:23 AM, Bryce = Godfrey wrote:

Both = .2 and .3 list the same from the mbean that Unreachable is empty = collection, and Live node lists all 3 nodes = still:
192.168.20.2
192.168.20.3
192.168.20.1

The = removetoken was done a few days ago, and I believe the remove was done = from .2

Here = is what ring outlook looks like, not sure why I get that token on the = empty first line = either:
Address =         DC =          Rack =        Status State   Load =            Owns =    Token
=             &n= bsp;           &nbs= p;            =             &n= bsp;           &nbs= p;            =  85070591730234615865843651857942052864
=
192.168.20.2 =    datacenter1 rack1 =       Up     Normal =  79.53 GB       50.00% =  0
192.168.20.3    datacenter1 rack1 =       Up     Normal =  42.63 GB       50.00% =  85070591730234615865843651857942052864
=

Yes, = both nodes show the same thing when doing a describe cluster, that .1 is = unreachable.


-----Original = Message-----
From: = aaron morton [mailto:aaron@thelastpickle.com] =
Sent: = Sunday, August 21, 2011 4:23 = AM
To: user@cassandra.apache.org
Subject:= Re: Completely removing a node from the = cluster

Unreachable nodes in either did not respond to the message = or were known to be down and were not sent a message. =
The = way the node lists are obtained for the ring command and describe = cluster are the same. So it's a bit odd. =

Can = you connect to JMX and have a look at the o.a.c.db.StorageService MBean = ? What do the LiveNode and UnrechableNodes attributes say ? =

Also = how long ago did you remove the token and on which machine? Do both 20.2 = and 20.3 think 20.1 is still around ? =

Cheers


-----------------
=
Aaron = Morton
Freelance Cassandra = Developer
@aaronmorton
http://www.thelastpickle.com
=

On = 20/08/2011, at 9:48 AM, Bryce Godfrey = wrote:

I'm on = 0.8.4

I have removed a dead node from = the cluster using nodetool removetoken command, and moved one of the = remaining nodes to rebalance the tokens.  Everything looks fine = when I run nodetool ring now, as it only lists the remaining 2 nodes and = they both look fine, owning 50% of the = tokens.

However, I can still see it = being considered as part of the cluster from the Cassandra-cli = (192.168.20.1 being the removed node) and I'm worried that the cluster = is still queuing up hints for the node, or any other issues it may = cause:

Cluster = Information:
Snitch: = org.apache.cassandra.locator.SimpleSnitch
Partitioner: = org.apache.cassandra.dht.RandomPartitioner
Schema = versions:
=    dcc8f680-caa4-11e0-0000-553d4dced3ff: [192.168.20.2, = 192.168.20.3]
   UNREACHABLE: = [192.168.20.1]


Do I need to do something else = to completely remove this = node?

Thanks,
Bryce





= --Apple-Mail=_B53BA189-4CA0-4543-B5B2-6B6E59797830--