Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 235AA9997 for ; Tue, 24 Jul 2012 03:25:14 +0000 (UTC) Received: (qmail 98094 invoked by uid 500); 24 Jul 2012 03:25:11 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 97800 invoked by uid 500); 24 Jul 2012 03:25:11 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 97765 invoked by uid 99); 24 Jul 2012 03:25:10 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Jul 2012 03:25:10 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of eran.chinthaka@gmail.com designates 209.85.220.172 as permitted sender) Received: from [209.85.220.172] (HELO mail-vc0-f172.google.com) (209.85.220.172) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Jul 2012 03:25:03 +0000 Received: by vcbfo14 with SMTP id fo14so5794968vcb.31 for ; Mon, 23 Jul 2012 20:24:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=AcjYQdt24YTkutN2GU+gazQA3qVYG/+BxyXowwZY0fU=; b=bFlYDgS2nKzIr5+faN3WEzY0gxzYvlZ5kJPH005POCtPP63soapDJ7yUAR21GlJlC0 78RE5HG14Bi7VAkI2Vfj4q47oeIz09RPW3JGRvoW5h38xdDJY0uXcvTnuy8kfXNQ5cNh 4MYiFjjfkRYwNWL2lVanDb5F9143s7T1luKVd2kBL0ZgrFd9kDrcJnkLEgzGtkomBmsX RufwWkGxAhzx9NthsXQAvMJt97L//Qg5787yfntkGY7c5IE9e4wHXdoxUmEkPf1z3lxY ALyUlExWMUD3nrZGyIuFPN1tm3GtWU6i7HndXAqk+bw07OTFoYVjMoXZc+th1Wh6HhtW ns5g== MIME-Version: 1.0 Received: by 10.52.28.71 with SMTP id z7mr12663716vdg.105.1343100282855; Mon, 23 Jul 2012 20:24:42 -0700 (PDT) Received: by 10.220.227.5 with HTTP; Mon, 23 Jul 2012 20:24:42 -0700 (PDT) In-Reply-To: References: Date: Mon, 23 Jul 2012 20:24:42 -0700 Message-ID: Subject: Re: Bringing a dead node back up after fixing hardware issues From: Eran Chinthaka Withana To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=20cf3078119e41e0ee04c58ae762 X-Virus-Checked: Checked by ClamAV on apache.org --20cf3078119e41e0ee04c58ae762 Content-Type: text/plain; charset=ISO-8859-1 Thanks Brandon for the answer (and I didn't know driftx = Brandon Williams. Thanks for your awesome support in Cassandra IRC) Increasing CL is tricky for us for now, as our RF on that datacenter is 2 and CL is set to ONE. If we make the CL to be LOCAL_QUORUM, then, if a node goes down we will have trouble. I will try to increase the RF to 3 in that data center and set the CL to LOCAL_QUORUM if nothing works out. About decommissioning, if the node goes down. There is no way of knowing running that command on that node, right? IIUC, decommissioning should be run on a node that needs to be decommissioned. Coming back to the original question, without touching the CL, can we bring back a dead node (after fixing it) and somehow tell Cassandra that the node is backup and do not send read requests until it gets all the data? Thanks, Eran Chinthaka Withana On Mon, Jul 23, 2012 at 6:48 PM, Brandon Williams wrote: > On Mon, Jul 23, 2012 at 6:26 PM, Eran Chinthaka Withana > wrote: > > Method 1: I copied the data from all the nodes in that data center, into > the > > repaired node, and brought it back up. But because of the rate of updates > > happening, the read misses started going up. > > That's not really a good method when you scale up and the amount of > data in the cluster won't fit on a single machine. > > > Method 2: I issued a removetoken command for that node's token and let > the > > cluster stream the data into relevant nodes. At the end of this process, > the > > dead node was not showing up in the ring output. Then I brought the node > > back up. I was expecting, Cassandra to first stream data into the new > node > > (which happens to be the dead node which was in the cluster earlier) and > > once its done then make it serve reads. But, in the server log, I can > see as > > soon the node comes up, it started serving reads, creating a large > number of > > read misses. > > Removetoken is for dead nodes, so the node has no way of locally > knowing it shouldn't be a cluster member any longer when it starts up. > Instead if you had decommissioned, it would have saved a flag to > indicate it should bootstrap at the next startup. > > > So the question is, what is the best way to bring back a dead node (once > its > > hardware issues are fixed) without impacting read misses? > > Increase your consistency level. Run a repair on the node once it's > back up, unless the repair time took longer than gc_grace, in which > case you need to removetoken it, delete all the data, and bootstrap it > back in if you don't want anything deleted to resurrect. > > -Brandon > --20cf3078119e41e0ee04c58ae762 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks Brandon for the answer (and I didn't know driftx =3D Brandon Wil= liams. Thanks for your awesome support in Cassandra IRC)

Increasing CL is tricky for us for now, as our RF on that datacenter is 2 = and CL is set to ONE. If we make the CL to be LOCAL_QUORUM, then, if a node= goes down we will have trouble. I will try to increase the RF to 3 in that= data center and set the CL to LOCAL_QUORUM if nothing works out.=A0

About decommissioning, if the node goes down. There is = no way of knowing running that command on that node, right? IIUC, decommiss= ioning should be run on a node that needs to be decommissioned.=A0

Coming back to the original question, without touching the C= L, can we bring back a dead node (after fixing it) and somehow tell Cassand= ra that the node is backup and do not send read requests until it gets all = the data?

Thanks,
Eran Chinthaka Withana


On Mon, Jul 23, 2012 at 6:48 PM, Brandon= Williams <driftx@gmail.com> wrote:
On Mon, Jul 23, 2012 at 6:26 PM, Eran Chinthaka Withana <eran.chinthaka@gmail.com> wrote:
> Method 1: I copied the data from all the nodes in that data center, in= to the
> repaired node, and brought it back up. But because of the rate of upda= tes
> happening, the read misses started going up.

That's not really a good method when you scale up and the amount = of
data in the cluster won't fit on a single machine.

> Method 2: I issued a removetoken command for that node's token and= let the
> cluster stream the data into relevant nodes. At the end of this proces= s, the
> dead node was not showing up in the ring output. Then I brought the no= de
> back up. I was expecting, Cassandra to first stream data into the new = node
> (which happens to be the dead node which was in the cluster earlier) a= nd
> once its done then make it serve reads. But, in the server log, I can = see as
> soon the node comes up, it started serving reads, creating a large num= ber of
> read misses.

Removetoken is for dead nodes, so the node has no way of locally
knowing it shouldn't be a cluster member any longer when it starts up.<= br> =A0Instead if you had decommissioned, it would have saved a flag to
indicate it should bootstrap at the next startup.

> So the question is, what is the best way to bring back a dead node (on= ce its
> hardware issues are fixed) without impacting read misses?

Increase your consistency level. =A0Run a repair on the node once it&= #39;s
back up, unless the repair time took longer than gc_grace, in which
case you need to removetoken it, delete all the data, and bootstrap it
back in if you don't want anything deleted to resurrect.

-Brandon

--20cf3078119e41e0ee04c58ae762--