Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 41DC3EAC0 for ; Tue, 19 Mar 2013 15:38:46 +0000 (UTC) Received: (qmail 23084 invoked by uid 500); 19 Mar 2013 15:38:43 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 23060 invoked by uid 500); 19 Mar 2013 15:38:43 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 23050 invoked by uid 99); 19 Mar 2013 15:38:43 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Mar 2013 15:38:43 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [192.174.58.133] (HELO XEDGEB.nrel.gov) (192.174.58.133) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Mar 2013 15:38:37 +0000 Received: from XHUBA.nrel.gov (10.20.4.58) by XEDGEB.nrel.gov (192.174.58.133) with Microsoft SMTP Server (TLS) id 8.3.245.1; Tue, 19 Mar 2013 09:38:13 -0600 Received: from MAILBOX2.nrel.gov ([fe80::19a0:6c19:6421:12f]) by XHUBA.nrel.gov ([::1]) with mapi; Tue, 19 Mar 2013 09:38:15 -0600 From: "Hiller, Dean" To: "user@cassandra.apache.org" Date: Tue, 19 Mar 2013 09:38:16 -0600 Subject: Re: Recovering from a faulty cassandra node Thread-Topic: Recovering from a faulty cassandra node Thread-Index: Ac4kt8XiYHmRjF4+Qc2ZEfATggQBLg== Message-ID: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.3.2.130206 acceptlanguage: en-US Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org Since you "cleared" out that node, it IS the replacement node. Dean From: Jabbar Azam > Reply-To: "user@cassandra.apache.org" > Date: Tuesday, March 19, 2013 9:29 AM To: "user@cassandra.apache.org" > Subject: Re: Recovering from a faulty cassandra node Hello Dean. I'm using vnodes so can't specify a token. In addition I can't follow the r= eplace node docs because I don't have a replacement node. On 19 March 2013 15:25, Hiller, Dean > wrote: I have not done this as of yet but from all that I have read your best opti= on is to follow the replace node documentation which I belive you need to 1. Have the token be the same BUT add 1 to it so it doesn't think it's th= e same computer 2. Have the bootstrap option set or something so streaming takes affect. I would however test that all out in QA to make sure it works and if you ha= ve QUOROM reads/writes a good part of that test would be to take node X dow= n after your node Y is back in the cluster to make sure reads/writes are wo= rking on the node you fixed=85..you just need to make sure node X shares on= e of the token ranges of node Y AND your writes/reads are in that token ran= ge. Dean From: Jabbar Azam >> Reply-To: "user@cassandra.apache.org>" >> Date: Tuesday, March 19, 2013 8:51 AM To: "user@cassandra.apache.org>" >> Subject: Recovering from a faulty cassandra node Hello, I am using Cassandra 1.2.2 on a 4 node test cluster with vnodes. I waited f= or over a week to insert lots of data into the cluster. During the end of t= he process one of the nodes had a hardware fault. I have fixed the hardware fault but the filing system on that node is corru= pt so I'll have to reinstall the OS and cassandra. I can think of two ways of reintegrating the host into the cluster 1) shrink the cluster to three nodes and add the node into the cluster 2) Add the node into the cluster without shrinking I'm not sure of the best approach to take and I'm not sure how to achieve e= ach step. Can anybody help? -- Thanks Jabbar Azam -- Thanks Jabbar Azam