cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Commit log fills up in less than a minute
Date Thu, 25 Aug 2011 21:47:54 GMT
Could you put together some information on this in a ticket and references this one https://issues.apache.org/jira/browse/CASSANDRA-3071

The short term fix is to disable HH. You will still get consistent reads. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 25/08/2011, at 3:22 AM, Anand Somani wrote:

> So I have looked at the cluster from 
> Cassandra-client - describe cluster => shows correctly - 3 nodes
> used the StorageService - JMX bean =>UnreachableNodes - shows 0
> 
> If all these show the correct ring state, why are hints being maintained, looks like
that is the only way to find out about "phantom" nodes.
> 
> On Wed, Aug 24, 2011 at 8:01 AM, Anand Somani <meatforums@gmail.com> wrote:
> So, I restarted the cluster (not rolling), but it is still maintaining hints for the
IP's that are no longer part of the ring. nodetool ring shows things correctly (as only 3
nodes). 
> When I check thru the jmx hintedhandoff manager, it shows it is maintaining the hints
for those non existent IP's. So the question is 
>  - How can I remove these IP permanently, so hints do not get saved?
>  - Not all nodes see the same list of IP's
> 
> 
> 
> 
> On Sun, Aug 21, 2011 at 3:10 PM, aaron morton <aaron@thelastpickle.com> wrote:
> Yup, you can check the what HH is doing via JMX.
> 
> there is a bug in 0.7 that can result in log files not been deleted https://issues.apache.org/jira/browse/CASSANDRA-2829
> 
> Cheers
>  
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 22/08/2011, at 4:56 AM, Anand Somani wrote:
> 
>> We have a lot of space on /data, and looks like it was flushing data fine from file
timestamps. 
>> 
>> We did have a bit of goofup with IP's when bringing up a down node (and the commit
files have been around since then). Wonder if that is what triggered it and we have a bunch
of hinted handoff's being backed up.
>> 
>> For hinted hand off - how do I check if the nodes are collecting hints ( I do have
it turned on). I noticed console bean HintedHandManager, is that the only way to find out.
>> 
>> On Sun, Aug 21, 2011 at 9:20 AM, Peter Schuller <peter.schuller@infidyne.com>
wrote:
>> > When does the actual commit-data file get deleted.
>> >
>> > The flush interval on all my memtables is 60 minutes
>> 
>> They *should* be getting deleted when they no longer contain any data
>> that has not been flushed to disk. Are flushes definitely still
>> happening? Is it possible flushing has started failing (e.g. out of
>> disk)?
>> 
>> The only way I can think of over nodes directly affecting the commit
>> log size on your node would be e.g. hinted handoff resulting in burst
>> of writes.
>> 
>> --
>> / Peter Schuller (@scode on twitter)
>> 
> 
> 
> 


Mime
View raw message