incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From William Oberman <>
Subject how to stop out of control compactions?
Date Mon, 01 Apr 2013 19:38:57 GMT
I'll skip the prelude, but I worked myself into a bit of a jam.  I'm
recovering now, but I want to double check if I'm thinking about things

Basically, I was in a state where a majority of my servers wanted to do
compactions, and rather large ones.  This was impacting my site
performance.  I tried nodetool stop COMPACTION.  I tried
setcompactionthroughput=1.  I tried restarting servers, but they'd restart
the compactions pretty much immediately on boot.

Then I realized that:
nodetool stop COMPACTION
only stopped running compactions, and then the compactions would re-enqueue
themselves rather quickly.

So, right now I have:
1.) scripts running on N-1 servers looping on "nodetool stop COMPACTION" in
a tight loop
2.) On the "Nth" server I've disabled gossip/thrift and turned up
setcompactionthroughput to 999
3.) When the Nth server completes, I pick from the remaining N-1 (well, I'm
still running the first compaction, which is going to take 12 more hours,
but that is the plan at least).

Does this make sense?  Other than the fact there was probably warning signs
that would have prevented me from getting into this state in the first
place? :-)


View raw message