Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 83E409A4A for ; Mon, 5 Dec 2011 00:52:33 +0000 (UTC) Received: (qmail 34024 invoked by uid 500); 5 Dec 2011 00:52:31 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 33995 invoked by uid 500); 5 Dec 2011 00:52:31 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 33987 invoked by uid 99); 5 Dec 2011 00:52:31 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Dec 2011 00:52:31 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of edlinuxguru@gmail.com designates 209.85.210.172 as permitted sender) Received: from [209.85.210.172] (HELO mail-iy0-f172.google.com) (209.85.210.172) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Dec 2011 00:52:24 +0000 Received: by iaek3 with SMTP id k3so5707506iae.31 for ; Sun, 04 Dec 2011 16:52:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=Z+rwkiagWITYiewqE0C5cC9AIpXFG3eUqdJ7SpdQ0/E=; b=h+Ulp6DQHpsPMVTSDBnC0a8wjn5JXCtRA8QQkV64PQC6NzcGyBAQAYlPW3leSDZBY+ m6X6YLoEXtbAM5skzEPJC3w63fixtUFxDzbXq2r+Cx/8rTdVP7F9k2qWm4Gi7xG7M68r PK+/O29F5xoX6cijxQhDqDOKxsVLVnYSIjE2s= MIME-Version: 1.0 Received: by 10.231.63.136 with SMTP id b8mr1788693ibi.33.1323046323598; Sun, 04 Dec 2011 16:52:03 -0800 (PST) Received: by 10.42.246.71 with HTTP; Sun, 4 Dec 2011 16:52:03 -0800 (PST) In-Reply-To: References: <4EDAAF7E.40502@bnl.gov> <4EDACE10.6010804@bnl.gov> <4EDBB7A9.9060604@bnl.gov> <4EDBE255.4080905@bnl.gov> <4EDC0783.90509@bnl.gov> Date: Sun, 4 Dec 2011 19:52:03 -0500 Message-ID: Subject: Re: Repair failure under 0.8.6 From: Edward Capriolo To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=000e0cd4b5b223b7a304b34dba0e X-Virus-Checked: Checked by ClamAV on apache.org --000e0cd4b5b223b7a304b34dba0e Content-Type: text/plain; charset=ISO-8859-1 You can say the min compaction threshold to 2 and the max Compaction Threshold to 3. If you have enough disk space for a few minor compaction this should free up some disk space. On Sun, Dec 4, 2011 at 7:17 PM, Peter Schuller wrote: > > As a side effect of the failed repair (so it seems) the disk usage on the > > affected node prevents compaction from working. It still works on > > the remaining nodes (we have 3 total). > > Is there a way to scrub the extraneous data? > > This is one of the reasons why killing an in-process repair is a bad thing > :( > > If you do not have enough disk space for any kind of compaction to > work, then no, unfortunately there is no easy way to get rid of the > data. > > You can go to extra trouble such as moving the entire node to some > other machine (e.g. firewalled from the cluster) with more disk and > run compaction there and then "move it back", but that is kind of > painful to do. Another option is to decommission the node and replace > it. However, be aware that (1) that leaves the ring with less capacity > for a while, and (2) when you decommission, the data you stream from > that node to others would be artificially inflated due to the repair > so there is some risk of "infecting" the other nodes with a large data > set. > > I should mention that if you have no traffic running against the > cluster, one way is to just remove all the data and then run repair > afterwards. But that implies that you're trusting that (1) no reads > are going to the cluster (else you might serve reads based on missing > data) and (2) that you are comfortable with loss of the data on the > node. (2) might be okay if you're e.g. writing at QUORUM at all times > and have RF >= 3 (basically, this is as if the node would have been > lost due to hardware breakage). > > A faster way to reconstruct the node would be to delete the data from > your keyspaces (except the system keyspace), start the node (now > missing data), and run 'nodetool rebuild' after > https://issues.apache.org/jira/browse/CASSANDRA-3483 is done. The > patch attached to that ticket should work for 0.8.6 I suspect (but no > guarantees). This also assumes you have no reads running against the > cluster. > > -- > / Peter Schuller (@scode, http://worldmodscode.wordpress.com) > --000e0cd4b5b223b7a304b34dba0e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable You can say the min compaction threshold to 2 and the max Compaction Thresh= old to 3. If you have enough disk space for a few minor compaction this sho= uld free up some disk space.

On Sun, Dec = 4, 2011 at 7:17 PM, Peter Schuller <peter.schuller@infidyne.com> wro= te:
> As a side ef= fect of the failed repair (so it seems) the disk usage on the
> affected node prevents compaction from working. It still works on
> the remaining nodes (we have 3 total).
> Is there a way to scrub the extraneous data?

This is one of the reasons why killing an in-process repair is a bad thing = :(

If you do not have enough disk space for any kind of compaction to
work, then no, unfortunately there is no easy way to get rid of the
data.

You can go to extra trouble such as moving the entire node to some
other machine (e.g. firewalled from the cluster) with more disk and
run compaction there and then "move it back", but that is kind of=
painful to do. Another option is to decommission the node and replace
it. However, be aware that (1) that leaves the ring with less capacity
for a while, and (2) when you decommission, the data you stream from
that node to others would be artificially inflated due to the repair
so there is some risk of "infecting" the other nodes with a large= data
set.

I should mention that if you have no traffic running against the
cluster, one way is to just remove all the data and then run repair
afterwards. But that implies that you're trusting that (1) no reads
are going to the cluster (else you might serve reads based on missing
data) and (2) that you are comfortable with loss of the data on the
node. (2) might be okay if you're e.g. writing at QUORUM at all times and have RF >=3D 3 (basically, this is as if the node would have been lost due to hardware breakage).

A faster way to reconstruct the node would be to delete the data from
your keyspaces (except the system keyspace), start the node (now
missing data), and run 'nodetool rebuild' after
https://issues.apache.org/jira/browse/CASSANDRA-3483 is done. T= he
patch attached to that ticket should work for 0.8.6 I suspect (but no
guarantees). This also assumes you have no reads running against the
cluster.

--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)

--000e0cd4b5b223b7a304b34dba0e--