Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Date: Sat, 10 Apr 2010 12:24:38 -0700
From: Anthony Molinaro <anthonym@alumni.caltech.edu>
To: user@cassandra.apache.org
Subject: Recovery from botched compaction
Message-ID: <20100410192438.GA82910@alumni.caltech.edu>
Mail-Followup-To: user@cassandra.apache.org
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.4.2.3i

Hi,

  This is sort of a pre-emptive question as the compaction I'm doing hasn't
failed yet but I expect it to any time now.  I have a cluster which has been
storing user profile data for a client.  Recently I've had to go back and
reload all the data again.  I wasn't watching diskspace, and on one of the
nodes it went above 50% (which I recall was bad), to somewhere around 70%.
I expect to most back with a compaction (as most of the data was the same
so a compaction should remove old copies), and went ahead and started one
with nodeprobe compact (using 0.5.0 on this cluster).  However, I do see
that the disk usage is growing (it's at 91% now).

So when the disk fills up and this compaction crashes what can I do?
I assume get a bigger disk, shut down the node, move the data and
restart will work, but do I have other options?
Which files can I ignore (ie, can I not move any of the *-tmp-* files)?
Will my system be in a corrupt state?

This machine is one in a set of 6, and since I didn't choose tokens
initially, they are very lopsided (ie, some use 20% of their disk, others
60-70%).  If I were to start moving tokens around would the machines short
of space be able to anti-compact without filling up?  or does anti-compaction
like compaction require 2x disk space?

Thanks,

-Anthony

-- 
------------------------------------------------------------------------
Anthony Molinaro                           <anthonym@alumni.caltech.edu>