incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Wright <kwri...@nanigans.com>
Subject Very large HintsColumnFamily
Date Sat, 22 Dec 2012 00:25:56 GMT
Hi all,

    I am seeing a VERY large HintsColumnFamily (40+ GB) on one of my nodes (I have 2 DC with
3 nodes each with 2 RF).  Nodetool ring as a result reports load as being way higher for the
one node (the delta being the size of the HintsColumnFamily).  This behavior seems to occur
if I do a large amount of data loading using that node as the coordinator node.  I found a
post related to this (http://mail-archives.apache.org/mod_mbox/cassandra-user/201203.mbox/%3C376CEC01195C894CB9F8A3C274029A96B471D61A@FISH-EX2K10-01.azaleos.net%3E)
but wanted to see if there were better ways to handle it then the reset suggested as it seems
somewhat risky.  Nodetool netstats never seems to show any streaming data.  With past nodes
it seemed like the node eventually fixed itself.  Note that I have the OOTB gc_grace_seconds
so perhaps I just need to wait 10 days before that runs again and the data gets deleted? 
Is there a way to change gc_grace_seconds outside Cassandra.yaml and thus save myself a node
restart?

Note that I am seeing severely degraded performance on this node when it attempts to compact
the HintsColumnFamily to the point where I had to set setcompactionthroughput to 999 to ensure
it doesn't run again (after which the node started serving requests much faster).

I appreciate the help!

Thanks

Mime
View raw message