cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Coli <rc...@eventbrite.com>
Subject Re: sstable usage doubles after repair
Date Thu, 20 Nov 2014 18:15:07 GMT
On Thu, Nov 20, 2014 at 8:36 AM, Stephane Legay <slegay@looplogic.com>
wrote:

> I upgraded a 2 node cluster with RF = 2  from 1.0.9 to 2.0.11. I did
> rolling upgrades and upgradesstables after each upgrade.
>

To be clear, did you go through 1.1, and 1.2, or did you go directly from
1.0 to 2.0?


> We then moved our data to new hardware by shutting down each node, moving
> data to new machine, and starting up with auto_bootstrap = false.
>

This should not be implicated, especially if you verified the upgraded
nodes came up with the same tokens they had before.


> When all was done I ran a repair. Data went from 250GB to 400 GB per node.
> A week later, I am doing another repair, data filling the 800GB drive on
> each machine. Huge compaction on each node, constantly.
>

How frequently had you been running repair in 1.0.9? How often do you
DELETE?


> Where should I go from here? Will scrubbing fix the issue?
>

I would inspect the newly created SSTables from a repair and see what they
contain. I would also look at log lines which indicate how many rows are
being repaired, with a special eye towards whether the number of rows
repaired each time you repair is decreasing.

Also note that repair in 2.0 is serial by default, you probably want the
old behavior, which you can get with "-par" flag.

=Rob
http://twitter.com/rcolidba

Mime
View raw message