incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <>
Subject Re: Commit log fills up in less than a minute
Date Thu, 25 Aug 2011 21:47:54 GMT
Could you put together some information on this in a ticket and references this one

The short term fix is to disable HH. You will still get consistent reads. 


Aaron Morton
Freelance Cassandra Developer

On 25/08/2011, at 3:22 AM, Anand Somani wrote:

> So I have looked at the cluster from 
> Cassandra-client - describe cluster => shows correctly - 3 nodes
> used the StorageService - JMX bean =>UnreachableNodes - shows 0
> If all these show the correct ring state, why are hints being maintained, looks like
that is the only way to find out about "phantom" nodes.
> On Wed, Aug 24, 2011 at 8:01 AM, Anand Somani <> wrote:
> So, I restarted the cluster (not rolling), but it is still maintaining hints for the
IP's that are no longer part of the ring. nodetool ring shows things correctly (as only 3
> When I check thru the jmx hintedhandoff manager, it shows it is maintaining the hints
for those non existent IP's. So the question is 
>  - How can I remove these IP permanently, so hints do not get saved?
>  - Not all nodes see the same list of IP's
> On Sun, Aug 21, 2011 at 3:10 PM, aaron morton <> wrote:
> Yup, you can check the what HH is doing via JMX.
> there is a bug in 0.7 that can result in log files not been deleted
> Cheers
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> On 22/08/2011, at 4:56 AM, Anand Somani wrote:
>> We have a lot of space on /data, and looks like it was flushing data fine from file
>> We did have a bit of goofup with IP's when bringing up a down node (and the commit
files have been around since then). Wonder if that is what triggered it and we have a bunch
of hinted handoff's being backed up.
>> For hinted hand off - how do I check if the nodes are collecting hints ( I do have
it turned on). I noticed console bean HintedHandManager, is that the only way to find out.
>> On Sun, Aug 21, 2011 at 9:20 AM, Peter Schuller <>
>> > When does the actual commit-data file get deleted.
>> >
>> > The flush interval on all my memtables is 60 minutes
>> They *should* be getting deleted when they no longer contain any data
>> that has not been flushed to disk. Are flushes definitely still
>> happening? Is it possible flushing has started failing (e.g. out of
>> disk)?
>> The only way I can think of over nodes directly affecting the commit
>> log size on your node would be e.g. hinted handoff resulting in burst
>> of writes.
>> --
>> / Peter Schuller (@scode on twitter)

View raw message