incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sylvain Lebresne <sylv...@datastax.com>
Subject Re: shutdown by drain
Date Fri, 23 Sep 2011 09:47:26 GMT
On Fri, Sep 23, 2011 at 11:04 AM, Radim Kolar <hsn@sendmail.cz> wrote:
> Dne 10.9.2011 21:48, Chris Goffinet napsal(a):
>>
>> For things like rolling restarts, we do:
>>
>> disablethrift
>> disablegossip
>> (...wait for all nodes to see this node go down..)
>> drain
>
> I discovered problem with this advice.
>
> If i do nodetool drain before killing node nodetool returns just after flush
> and stuff disabling is finished on cassandra node. But flush can trigger
> possible compaction and if you kill node after drain it will interrupt
> compaction in progress resulting in wasted disk space. I am not sure if tmp
> files are cleaned on cassandra start.

While it's true that flush/drain may trigger compactions and killing the node
just after them will interrupt those compaction, it's not really a problem of
flush or drain, in that you always have the risk of interrupting a compaction
when killing a node (i.e, even if drain don't happen to trigger a compaction,
there may have a compaction that started before the drain and are not yet
finished when you kill the node). Besides, if you kill the node just after the
drain, the compaction that it has trigger are probably not very advanced yet
so are probably the compaction that are the less wasteful to interrupt.

Moreover, yes, tmp files are cleaned on restart. So for a rolling restart,
which was the case Chris was talking about, this can hardly be called
wasted space.

Now it is true that it could be a shame to interrupt a compaction that have
been running for a long time and is about to finish (so typically not one that
has just been triggered by your drain), but you can always check the
compaction manager in JMX to see if it's the case before killing the node.

--
Sylvain

Mime
View raw message