cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremy Hanna <jeremy.hanna1...@gmail.com>
Subject Re: hints system CF getting out of control
Date Fri, 19 Aug 2011 06:15:15 GMT
So if you kill a joining node, it will remain as a Fat Client (as of 0.7.x) in gossip.  That
is innocuous on its own.  However if a node was decommissioned while that was joining, then
the lingering Fat Client will also gossip about the node that was decommissioned.  Then it
will try to hint for that decommissioned node.  That was the origin of our out of control
hints.

So we killed the joining node, cleared it out to start fresh, and did a full cluster restart
to clear out the gossip of the Fat Clients and the hints.  That worked.

On Aug 18, 2011, at 7:56 PM, Jonathan Ellis wrote:

> I would assume it's because it thinks some node is down and is
> creating hints for it.
> 
> On Thu, Aug 18, 2011 at 6:31 PM, Jeremy Hanna
> <jeremy.hanna1234@gmail.com> wrote:
>> We're trying to bootstrap some new nodes and it appears when adding a new node that
there is a lot of logging on hints being flushed and compacted.  It's been taking about 75
minutes thus far to bootstrap for only about 10 GB of data.  It's ballooned up to over 40
GB on the new node.  I do 'ls -l $CASSANDRA/data/system | grep -i hint | wc -l' and it's over
2000.
>> 
>> Any idea why this would be happening and why it's going crazy flushing these files?
 It's also trying to keep up and compacting them as well.
>> 
>> Completed flushing $CASSANDRA/data/system/HintsColumnFamily-f-1740-Data.db
>> 
>> Version 0.7.8 (upgrading from 0.7.6 right now)
>> Number of nodes: 20
> 
> 
> 
> -- 
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com


Mime
View raw message