incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anand Somani <meatfor...@gmail.com>
Subject Re: Commit log fills up in less than a minute
Date Fri, 26 Aug 2011 15:23:27 GMT
Sure I can fill in the ticket. Here is what I have noticed so far, the count
of HH is not going up, which is good. I think what must have happened is
that after I restarted the cluster, no new hints were added just the old
one's are still around and not cleaned up, is that possible? Cannot say for
sure, since I only looked at this JMX bean abt 36 hours after restart.

Can I just clean this up using the JMX call? I do not want to turn off HH,
since that can handle the intermittent network hiccups well, right?

On Thu, Aug 25, 2011 at 2:47 PM, aaron morton <aaron@thelastpickle.com>wrote:

> Could you put together some information on this in a ticket and references
> this one https://issues.apache.org/jira/browse/CASSANDRA-3071
>
> The short term fix is to disable HH. You will still get consistent reads.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 25/08/2011, at 3:22 AM, Anand Somani wrote:
>
> So I have looked at the cluster from
>
>    - Cassandra-client - describe cluster => shows correctly - 3 nodes
>    - used the StorageService - JMX bean =>UnreachableNodes - shows 0
>
>
> If all these show the correct ring state, why are hints being maintained,
> looks like that is the only way to find out about "phantom" nodes.
>
> On Wed, Aug 24, 2011 at 8:01 AM, Anand Somani <meatforums@gmail.com>wrote:
>
>> So, I restarted the cluster (not rolling), but it is still maintaining
>> hints for the IP's that are no longer part of the ring. nodetool ring shows
>> things correctly (as only 3 nodes).
>> When I check thru the jmx hintedhandoff manager, it shows it is
>> maintaining the hints for those non existent IP's. So the question is
>>  - How can I remove these IP permanently, so hints do not get saved?
>>  - Not all nodes see the same list of IP's
>>
>>
>>
>>
>> On Sun, Aug 21, 2011 at 3:10 PM, aaron morton <aaron@thelastpickle.com>wrote:
>>
>>> Yup, you can check the what HH is doing via JMX.
>>>
>>> there is a bug in 0.7 that can result in log files not been deleted
>>> https://issues.apache.org/jira/browse/CASSANDRA-2829
>>>
>>> Cheers
>>>
>>>  -----------------
>>> Aaron Morton
>>> Freelance Cassandra Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 22/08/2011, at 4:56 AM, Anand Somani wrote:
>>>
>>> We have a lot of space on /data, and looks like it was flushing data fine
>>> from file timestamps.
>>>
>>> We did have a bit of goofup with IP's when bringing up a down node (and
>>> the commit files have been around since then). Wonder if that is what
>>> triggered it and we have a bunch of hinted handoff's being backed up.
>>>
>>> For hinted hand off - how do I check if the nodes are collecting hints (
>>> I do have it turned on). I noticed console bean HintedHandManager, is that
>>> the only way to find out.
>>>
>>> On Sun, Aug 21, 2011 at 9:20 AM, Peter Schuller <
>>> peter.schuller@infidyne.com> wrote:
>>>
>>>> > When does the actual commit-data file get deleted.
>>>> >
>>>> > The flush interval on all my memtables is 60 minutes
>>>>
>>>> They *should* be getting deleted when they no longer contain any data
>>>> that has not been flushed to disk. Are flushes definitely still
>>>> happening? Is it possible flushing has started failing (e.g. out of
>>>> disk)?
>>>>
>>>> The only way I can think of over nodes directly affecting the commit
>>>> log size on your node would be e.g. hinted handoff resulting in burst
>>>> of writes.
>>>>
>>>> --
>>>> / Peter Schuller (@scode on twitter)
>>>>
>>>
>>>
>>>
>>
>
>

Mime
View raw message