Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 328746E5B for ; Thu, 26 May 2011 03:28:45 +0000 (UTC) Received: (qmail 84554 invoked by uid 500); 26 May 2011 03:28:43 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 84489 invoked by uid 500); 26 May 2011 03:28:42 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 84481 invoked by uid 99); 26 May 2011 03:28:41 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 03:28:41 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jonathan.colby@gmail.com designates 74.125.82.198 as permitted sender) Received: from [74.125.82.198] (HELO mail-wy0-f198.google.com) (74.125.82.198) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 03:28:35 +0000 Received: by wyf19 with SMTP id 19so91649wyf.1 for ; Wed, 25 May 2011 20:28:14 -0700 (PDT) MIME-Version: 1.0 Received: by 10.216.58.147 with SMTP id q19mr411472wec.11.1306380493942; Wed, 25 May 2011 20:28:13 -0700 (PDT) In-Reply-To: <18A1675F-269A-4688-A60C-981FEF62A879@thelastpickle.com> Message-ID: <000e0cdff81c4879f504a42569ab@google.com> Date: Thu, 26 May 2011 03:28:13 +0000 Subject: Re: Re: nodetool move trying to stream data to node no longer in cluster From: jonathan.colby@gmail.com To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=000e0cdff81c4879c804a42569a8 --000e0cdff81c4879c804a42569a8 Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes Content-Transfer-Encoding: base64 U2VlbXMgbGlrZSBpdCBoYWQgc29tZXRoaW5nIHRvIGRvIHdpdGggc3RhbGUgZW5kcG9pbnQgaW5m b3JtYXRpb24uIEkgZGlkIGEgIA0Kcm9sbGluZyByZXN0YXJ0IG9mIHRoZSB3aG9sZSBjbHVzdGVy IGFuZCB0aGF0IHNlZW1lZCB0byB0cmlnZ2VyIHRoZSBub2RlcyAgDQp0byByZW1vdmUgdGhlIG5v ZGUgdGhhdCB3YXMgZGVjb21taXNzaW9uZWQuDQoNCk9uICwgYWFyb24gbW9ydG9uIDxhYXJvbkB0 aGVsYXN0cGlja2xlLmNvbT4gd3JvdGU6DQo+IElzIGl0IHNob3dpbmcgcHJvZ3Jlc3MgPyBJdCBt YXkganVzdCBiZSBhIHByb2JsZW0gd2l0aCB0aGUgaW5mb3JtYXRpb24gIA0KPiBwcmludGVkIG91 dC4NCg0KDQoNCj4gQ2FuIHlvdSBjaGVjayBmcm9tIHRoZSBvdGhlciBub2RlcyBpbiB0aGUgY2x1 c3RlciB0byBzZWUgaWYgdGhleSBhcmUgIA0KPiByZWNlaXZpbmcgdGhlIHN0cmVhbSA/DQoNCg0K DQo+IGNoZWVycw0KDQoNCg0KPiAtLS0tLS0tLS0tLS0tLS0tLQ0KDQo+IEFhcm9uIE1vcnRvbg0K DQo+IEZyZWVsYW5jZSBDYXNzYW5kcmEgRGV2ZWxvcGVyDQoNCj4gQGFhcm9ubW9ydG9uDQoNCj4g aHR0cDovL3d3dy50aGVsYXN0cGlja2xlLmNvbQ0KDQoNCg0KPiBPbiAyNiBNYXkgMjAxMSwgYXQg MDA6NDIsIEpvbmF0aGFuIENvbGJ5IHdyb3RlOg0KDQoNCg0KPiA+IEkgcmVjZW50bHkgcmVtb3Zl ZCBhIG5vZGUgKHdpdGggZGVjb21taXNzaW9uKSBmcm9tIG91ciBjbHVzdGVyLg0KDQo+ID4NCg0K PiA+IEkgYWRkZWQgYSBjb3VwbGUgbmV3IG5vZGVzIGFuZCBhbSBub3cgdHJ5aW5nIHRvIHJlYmFs YW5jZSB0aGUgY2x1c3RlciAgDQo+IHVzaW5nIG5vZGV0b29sIG1vdmUuDQoNCj4gPg0KDQo+ID4g SG93ZXZlciwgbmV0c3RhdHMgc2hvd3MgdGhhdCB0aGUgbm9kZSBiZWluZyAibW92ZWQiIGlzIHRy eWluZyB0byBzdHJlYW0gIA0KPiBkYXRhIHRvIHRoZSBub2RlIHRoYXQgSSBhbHJlYWR5IGRlY29t bWlzc2lvbmVkIHllc3RlcmRheS4NCg0KPiA+DQoNCj4gPiBUaGUgcmVtb3ZlZCBub2RlIHdhcyBw b3dlcmVkLW9mZiwgdGFrZW4gb3V0IG9mIGRucywgaXRzIElQIGlzIG5vdCBldmVuICANCj4gcGlu Z2FibGUuIEl0IHdhcyBuZXZlciBhIHNlZWQgbmVpdGhlci4NCg0KPiA+DQoNCj4gPiBUaGlzIGlz IGNhc3NhbmRyYSAwLjcuNSBvbiA2NGJpdCBsaW51eC4gSG93IGRvIEkgdGVsbCB0aGUgY2x1c3Rl ciB0aGF0ICANCj4gdGhpcyBub2RlIGlzIGdvbmU/IEdvc3NpcCBzaG91bGQgaGF2ZSBkZXRlY3Rl ZCB0aGlzLiBUaGUgcmluZyBjb21tYW5kcyAgDQo+IHNob3dzIHRoZSBjb3JyZWN0IGNsdXN0ZXIg SVBzLg0KDQo+ID4NCg0KPiA+IEhlcmUgaXMgYSBwb3J0aW9uIG9mIG5ldHN0YXRzLiAxMC40Ni4x MDguMTAyIGlzIHRoZSBub2RlIHdoaWNoIHdhcyAgDQo+IHJlbW92ZWQuDQoNCj4gPg0KDQo+ID4g TW9kZTogTGVhdmluZzogc3RyZWFtaW5nIGRhdGEgdG8gb3RoZXIgbm9kZXMNCg0KPiA+IFN0cmVh bWluZyB0bzogLzEwLjQ2LjEwOC4xMDINCg0KPiA+ICANCj4gL3Zhci9saWIvY2Fzc2FuZHJhL2Rh dGEvREZTL21haW4tZi0xMDY0LURhdGEuZGIvKDQ2ODEwMjcsNTE5NTQ5MSksKDUxOTU0OTEsMTUz MDg1NzApLCgxNTMwODU3MCwxNTg5MTcxMCksKDE2MzM2NzUwLDIwNTU4NzA1KSwoMjA1NTg3MDUs MjkxMTIyMDMpLCgyOTExMjIwMywzNjI3OTMyOSksKDM2NDY1OTQyLDM2NjIzMjIzKSwoMzY3NDA0 NTcsMzcyMjcwNTgpLCgzNzIyNzA1OCw0MjIwNjk5NCksKDQyMjA2OTk0LDQ3MzgwMjk0KSwoNDc2 MzUwNTMsNDc3MDk4MTMpLCg0NzcwOTgxMyw0ODM1Mzk0NCksKDQ4NjIxMjg3LDQ5NDA2NDk5KSwo NTMzMzAwNDgsNTM1NzEzMTIpLCg1MzU3MTMxMiw1NDE1MzkyMiksKDU0MTUzOTIyLDU5ODU3NjE1 KSwoNTk4NTc2MTUsNjEwMjk5MTApLCg2MTAyOTkxMCw2MTg3MTUwOSksKDYyMTkwODAwLDYyNDk4 NjA1KSwoNjI4MjQyODEsNjI5NjQ4MzApLCg2MzUxMTYwNCw2NDM1MzExNCksKDY0MzUzMTE0LDY0 NzYwNDAwKSwoNjUxNzQ3MDIsNjU5MTk3NzEpLCg2NTkxOTc3MSw2NjQzNTYzMCksKDgxNDQwMDI5 LDgxNzI1OTQ5KSwoODE3MjU5NDksODMzMTM4NDcpLCg4MzMxMzg0Nyw4MzkwODcwOSksKDg4OTgz ODYzLDg5MjM3MzAzKSwoODkyMzczMDMsODk5MzQxOTkpLCg4OTkzNDE5OSw5Nw0KDQo+ID4gLi4u Li4uLi4uLi4uLi4uLi4uLg0KDQo+ID4gIA0KPiA1NjkzNDkxLDE0Nzk1ODYxNjY2KSwoMTQ3OTU4 NjE2NjYsMTQ3OTYxMDUzMTgpLCgxNDc5NjEwNTMxOCwxNDc5NjM2Njg4NiksKDE0Nzk2Njk5ODI1 LDE0ODAzODc0OTQxKSwoMTQ4MDM4NzQ5NDEsMTQ4MDg4OTgzMzEpLCgxNDgwODg5ODMzMSwxNDgx MTY3MDY5OSksKDE0ODExNjcwNjk5LDE0ODE1MTI1MTc3KSwoMTQ4MTUxMjUxNzcsMTQ4MTk3NjUw MDMpLCgxNDgyMDIyOTQzMywxNDgyMDg1ODI2NikNCg0KPiA+IHByb2dyZXNzPTI4MDU3NDM3NjQw Mi8xMjQzNDA0OTkwMCAtIDIyNTYlDQoNCj4gPiAuLi4uLg0KDQo+ID4NCg0KPiA+DQoNCj4gPiBO b3RlIDEwLjQ2LjEwOC4xMDIgaXMgTk9UIHBhcnQgb2YgdGhlIHJpbmcuDQoNCj4gPg0KDQo+ID4g QWRkcmVzcyBTdGF0dXMgU3RhdGUgTG9hZCBPd25zIFRva2VuDQoNCj4gPiAxNDg4NzM1MzU1Mjc5 MTA1Nzc3NjUyMjYzOTA3NTEzOTg1OTI1MTINCg0KPiA+IDEwLjQ2LjEwOC4xMDAgVXAgTm9ybWFs IDcxLjczIEdCIDEyLjUwJSAwDQoNCj4gPiAxMC40Ni4xMDguMTAxIFVwIE5vcm1hbCAxMDkuNjkg R0IgMTIuNTAlICANCj4gMjEyNjc2NDc5MzI1NTg2NTM5NjY0NjA5MTI5NjQ0ODU1MTMyMTYNCg0K PiA+IDEwLjQ3LjEwOC4xMDAgVXAgTGVhdmluZyAyODEuOTUgR0IgMzcuNTAlICANCj4gODUwNzA1 OTE3MzAyMzQ2MTU4NjU4NDM2NTE4NTc5NDIwNTI4NjMNCj4gPiAxMC40Ny4xMDguMTAyIFVwIE5v cm1hbCAyMTAuNzcgR0IgMC4wMCUgIA0KPiA4NTA3MDU5MTczMDIzNDYxNTg2NTg0MzY1MTg1Nzk0 MjA1Mjg2NA0KDQo+ID4gMTAuNDcuMTA4LjEwMSBVcCBOb3JtYWwgMjg5LjU5IEdCIDE2LjY3JSAg DQo+IDExMzQyNzQ1NTY0MDMxMjgyMTE1NDQ1ODIwMjQ3NzI1NjA3MDQ4NA0KDQo+ID4gMTAuNDYu MTA4LjEwMyBVcCBOb3JtYWwgMjk5Ljg3IEdCIDguMzMlICANCj4gMTI3NjA1ODg3NTk1MzUxOTIz Nzk4NzY1NDc3Nzg2OTEzMDc5Mjk2DQoNCj4gPiAxMC40Ny4xMDguMTAzIFVwIE5vcm1hbCA5NC45 OSBHQiAxMi41MCUgIA0KPiAxNDg4NzM1MzU1Mjc5MTA1Nzc3NjUyMjYzOTA3NTEzOTg1OTI1MTEN Cg0KPiA+IDEwLjQ2LjEwOC4xMDQgVXAgTm9ybWFsIDEwMy4wMSBHQiAwLjAwJSAgDQo+IDE0ODg3 MzUzNTUyNzkxMDU3Nzc2NTIyNjM5MDc1MTM5ODU5MjUxMg0KDQo+ID4NCg0KPiA+DQoNCj4gPg0K DQoNCg0K --000e0cdff81c4879c804a42569a8 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Seems like it had something to do with stale endpoint information. I did a= rolling restart of the whole cluster and that seemed to trigger the nodes = to remove the node that was decommissioned.

On , aaron morton &l= t;aaron@thelastpickle.com> wrote:
> Is it showing progress ? It = may just be a problem with the information printed out.
>
>= ;
>
> Can you check from the other nodes in the cluster t= o see if they are receiving the stream ?
>
>
> > cheers
>
>
>
> ----------------= -
>
> Aaron Morton
>
> Freelance Cassandr= a Developer
>
> @aaronmorton
>
> http://w= ww.thelastpickle.com
>
>
>
> On 26 May 2= 011, at 00:42, Jonathan Colby wrote:
>
>
>
= > > I recently removed a node (with decommission) from our cluster.>
> >
>
> > I added a couple new nod= es and am now trying to rebalance the cluster using nodetool move.
>= ;
> >
>
> > However, =A0netstats shows that= the node being "moved" is trying to stream data to the node that= I already decommissioned yesterday.
>
> >
> > > The removed node was powered-off, taken out of dns, its IP is= not even pingable. =A0 It was never a seed neither.
>
> &= gt;
>
> > This is cassandra 0.7.5 on 64bit linux. =A0 H= ow do I tell the cluster that this node is gone? =A0Gossip should have dete= cted this. =A0The ring commands shows the correct cluster IPs.
> > >
>
> > Here is a portion of netstats. 10.4= 6.108.102 is the node which was removed.
>
> >
>= ;
> > Mode: Leaving: streaming data to other nodes
> > > Streaming to: /10.46.108.102
>
> > =A0 /v= ar/lib/cassandra/data/DFS/main-f-1064-Data.db/(4681027,5195491),(5195491,15= 308570),(15308570,15891710),(16336750,20558705),(20558705,29112203),(291122= 03,36279329),(36465942,36623223),(36740457,37227058),(37227058,42206994),(4= 2206994,47380294),(47635053,47709813),(47709813,48353944),(48621287,4940649= 9),(53330048,53571312),(53571312,54153922),(54153922,59857615),(59857615,61= 029910),(61029910,61871509),(62190800,62498605),(62824281,62964830),(635116= 04,64353114),(64353114,64760400),(65174702,65919771),(65919771,66435630),(8= 1440029,81725949),(81725949,83313847),(83313847,83908709),(88983863,8923730= 3),(89237303,89934199),(89934199,97
>
> > .............= ......
>
> > 5693491,14795861666),(14795861666,14796105= 318),(14796105318,14796366886),(14796699825,14803874941),(14803874941,14808= 898331),(14808898331,14811670699),(14811670699,14815125177),(14815125177,14= 819765003),(14820229433,14820858266)
>
> > =A0 =A0 =A0 = =A0 progress=3D280574376402/12434049900 - 2256%
>
> > .= ....
>
> >
>
> >
>
&g= t; > Note 10.46.108.102 is NOT part of the ring.
>
> &g= t;
>
> > Address =A0 =A0 =A0 =A0 Status State =A0 Load = =A0 =A0 =A0 =A0 =A0 =A0Owns =A0 =A0Token
>
> > =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 148873535527910577765226390751398592512
> =
> > 10.46.108.100 =A0 Up =A0 =A0 Normal =A071.73 GB =A0 =A0 =A0= =A012.50% =A00
>
> > 10.46.108.101 =A0 Up =A0 =A0 Norm= al =A0109.69 GB =A0 =A0 =A0 12.50% =A02126764793255865396646091296448551321= 6
>
> > 10.47.108.100 =A0 Up =A0 =A0 Leaving 281.95 GB = =A0 =A0 =A0 37.50% =A085070591730234615865843651857942052863 =A0 =A0 =A0 > > 10.47.108.102 =A0 Up =A0 =A0 Normal =A0210.77 GB =A0 =A0 =A0 = 0.00% =A0 85070591730234615865843651857942052864
>
> > = 10.47.108.101 =A0 Up =A0 =A0 Normal =A0289.59 GB =A0 =A0 =A0 16.67% =A01134= 27455640312821154458202477256070484
>
> > 10.46.108.103= =A0 Up =A0 =A0 Normal =A0299.87 GB =A0 =A0 =A0 8.33% =A0 12760588759535192= 3798765477786913079296
>
> > 10.47.108.103 =A0 Up =A0 = =A0 Normal =A094.99 GB =A0 =A0 =A0 =A012.50% =A0148873535527910577765226390= 751398592511
>
> > 10.46.108.104 =A0 Up =A0 =A0 Normal = =A0103.01 GB =A0 =A0 =A0 0.00% =A0 148873535527910577765226390751398592512<= br />>
> >
>
> >
>
> &g= t;
>
>
> --000e0cdff81c4879c804a42569a8--