Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
Sender: scode@scode.org
In-Reply-To: 
 <CAG00uybf1iroC8yUD2By0G2CUaxUj3effJ9u3dFxmGXvQZNO2w@mail.gmail.com>
References: 
 <CAG00uybf1iroC8yUD2By0G2CUaxUj3effJ9u3dFxmGXvQZNO2w@mail.gmail.com>
Date: Fri, 19 Aug 2011 20:26:23 +0200
Message-ID: 
 <CAO5xsd2fPoZiMmopUVXXD45HuggLssDOFrD16_syc1LjeK4_MA@mail.gmail.com>
Subject: Re: nodetool repair caused high disk space usage
From: Peter Schuller <peter.schuller@infidyne.com>
To: user@cassandra.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

> After upgrading to cass 0.8.4 from cass 0.6.11.=C2=A0 I ran scrub.=C2=A0 =
That worked
> fine.=C2=A0 Then I ran nodetool repair on one of the nodes.=C2=A0 The dis=
k usage on
> data directory increased from 40GB to 480GB, and it's still growing.

If you check your data directory, does it contain a lot of
"*Compacted" files? It sounds like you're churning sstables from a
combination of compactions/flushes (including triggered by repair) and
the old ones aren't being deleted. I wonder if there is still some
issue causing sstable retention

Since you're on 0.8.4, I'm a bit suspicious. I'd have to re-check each
JIRA but I think the major known repair problems should be fixed
except for CASSANDRA-2280 which is not your problem since you're going
form a total load of 40  gig to hundreds of gigs (so even with all
cf:s streaming, that's unexpected).

Do you have any old left-over streams active on the nodes? "nodetool
netstats". If there are "stuck" streams, they might be causing sstable
retention beyond what you'd expect.

--=20
/ Peter Schuller (@scode on twitter)