incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anand Somani <meatfor...@gmail.com>
Subject Re: Commit log fills up in less than a minute
Date Wed, 24 Aug 2011 15:01:11 GMT
So, I restarted the cluster (not rolling), but it is still maintaining hints
for the IP's that are no longer part of the ring. nodetool ring shows things
correctly (as only 3 nodes).
When I check thru the jmx hintedhandoff manager, it shows it is maintaining
the hints for those non existent IP's. So the question is
 - How can I remove these IP permanently, so hints do not get saved?
 - Not all nodes see the same list of IP's



On Sun, Aug 21, 2011 at 3:10 PM, aaron morton <aaron@thelastpickle.com>wrote:

> Yup, you can check the what HH is doing via JMX.
>
> there is a bug in 0.7 that can result in log files not been deleted
> https://issues.apache.org/jira/browse/CASSANDRA-2829
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 22/08/2011, at 4:56 AM, Anand Somani wrote:
>
> We have a lot of space on /data, and looks like it was flushing data fine
> from file timestamps.
>
> We did have a bit of goofup with IP's when bringing up a down node (and the
> commit files have been around since then). Wonder if that is what triggered
> it and we have a bunch of hinted handoff's being backed up.
>
> For hinted hand off - how do I check if the nodes are collecting hints ( I
> do have it turned on). I noticed console bean HintedHandManager, is that the
> only way to find out.
>
> On Sun, Aug 21, 2011 at 9:20 AM, Peter Schuller <
> peter.schuller@infidyne.com> wrote:
>
>> > When does the actual commit-data file get deleted.
>> >
>> > The flush interval on all my memtables is 60 minutes
>>
>> They *should* be getting deleted when they no longer contain any data
>> that has not been flushed to disk. Are flushes definitely still
>> happening? Is it possible flushing has started failing (e.g. out of
>> disk)?
>>
>> The only way I can think of over nodes directly affecting the commit
>> log size on your node would be e.g. hinted handoff resulting in burst
>> of writes.
>>
>> --
>> / Peter Schuller (@scode on twitter)
>>
>
>
>

Mime
View raw message