Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F07317968 for ; Fri, 19 Aug 2011 18:42:29 +0000 (UTC) Received: (qmail 23990 invoked by uid 500); 19 Aug 2011 18:42:28 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 23885 invoked by uid 500); 19 Aug 2011 18:42:27 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 23877 invoked by uid 99); 19 Aug 2011 18:42:27 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Aug 2011 18:42:27 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.213.44] (HELO mail-yw0-f44.google.com) (209.85.213.44) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Aug 2011 18:42:22 +0000 Received: by ywm21 with SMTP id 21so2584878ywm.31 for ; Fri, 19 Aug 2011 11:42:01 -0700 (PDT) MIME-Version: 1.0 Received: by 10.143.79.17 with SMTP id g17mr43512wfl.22.1313779320728; Fri, 19 Aug 2011 11:42:00 -0700 (PDT) Received: by 10.143.9.4 with HTTP; Fri, 19 Aug 2011 11:42:00 -0700 (PDT) In-Reply-To: References: Date: Fri, 19 Aug 2011 14:42:00 -0400 Message-ID: Subject: Re: nodetool repair caused high disk space usage From: Huy Le To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001517401870b9ae1d04aae01569 --001517401870b9ae1d04aae01569 Content-Type: text/plain; charset=ISO-8859-1 There were few Compacted files. I thought that might have been the cause, but it wasn't it. We have a CF that is 23GB, and while repair is running, there are multiple instances of that CF created along with other CFs. I checked the stream directory across cluster of four nodes, but it was empty. I can not reproduce this issue in version 0.6.11 with a copy of data backed up prior to 0.8.4 upgrade. Currently repair is still running on two 0.6.11 nodes. My plan is to run compact across the cluster running 0.6.11. When done, make another attempt to upgrade it to 0.8.4. Huy On Fri, Aug 19, 2011 at 2:26 PM, Peter Schuller wrote: > > After upgrading to cass 0.8.4 from cass 0.6.11. I ran scrub. That > worked > > fine. Then I ran nodetool repair on one of the nodes. The disk usage on > > data directory increased from 40GB to 480GB, and it's still growing. > > If you check your data directory, does it contain a lot of > "*Compacted" files? It sounds like you're churning sstables from a > combination of compactions/flushes (including triggered by repair) and > the old ones aren't being deleted. I wonder if there is still some > issue causing sstable retention > > Since you're on 0.8.4, I'm a bit suspicious. I'd have to re-check each > JIRA but I think the major known repair problems should be fixed > except for CASSANDRA-2280 which is not your problem since you're going > form a total load of 40 gig to hundreds of gigs (so even with all > cf:s streaming, that's unexpected). > > Do you have any old left-over streams active on the nodes? "nodetool > netstats". If there are "stuck" streams, they might be causing sstable > retention beyond what you'd expect. > > -- > / Peter Schuller (@scode on twitter) > -- Huy Le Spring Partners, Inc. http://springpadit.com --001517401870b9ae1d04aae01569 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable There were few Compacted files.=A0 I thought that might have been the cause= , but it wasn't it.=A0 We have a CF that is 23GB, and while repair is r= unning, there are multiple instances of that CF created along with other CF= s.

I checked the stream directory across cluster of four nodes, but it was= empty.=A0

I can not reproduce this issue in version 0.6.11 with a = copy of data backed up prior to 0.8.4 upgrade.=A0

Currently repair = is still running on two 0.6.11 nodes.=A0 My plan is to run compact across t= he cluster running 0.6.11.=A0 When done, make another attempt to upgrade it= to 0.8.4.

Huy

On Fri, Aug 19, 2011 at 2:26 PM, = Peter Schuller <peter.schuller@infidyne.com> wrote:
> After upgrading to cass 0.8.4 from cass 0.6.11.=A0 I= ran scrub.=A0 That worked
> fine.=A0 Then I ran nodetool repair on one of the nodes.=A0 The disk u= sage on
> data directory increased from 40GB to 480GB, and it's still growin= g.

If you check your data directory, does it contain a lot of
"*Compacted" files? It sounds like you're churning sstables f= rom a
combination of compactions/flushes (including triggered by repair) and
the old ones aren't being deleted. I wonder if there is still some
issue causing sstable retention

Since you're on 0.8.4, I'm a bit suspicious. I'd have to re-che= ck each
JIRA but I think the major known repair problems should be fixed
except for CASSANDRA-2280 which is not your problem since you're going<= br> form a total load of 40 =A0gig to hundreds of gigs (so even with all
cf:s streaming, that's unexpected).

Do you have any old left-over streams active on the nodes? "nodetool netstats". If there are "stuck" streams, they might be causi= ng sstable
retention beyond what you'd expect.

--
/ Peter Schuller (@scode on twitter)



--
Huy Le
Sprin= g Partners, Inc.
http://springpadit.c= om
--001517401870b9ae1d04aae01569--