incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: 1.1 not removing commit log files?
Date Fri, 01 Jun 2012 02:01:03 GMT
Could be this 
https://issues.apache.org/jira/browse/CASSANDRA-4201

But that talks about segments not being cleared at startup. Does not explain why they were
allowed to get past the limit in the first place. 

Can you share some logs from the time the commit log got out of control ? 

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 1/06/2012, at 9:34 AM, Bryce Godfrey wrote:

> So this happened to me again, but it was only when the cluster had a node down for a
while.  Then the commit logs started piling up past the limit I set in the config file, and
filled the drive. 
> After the node recovered and hints had replayed the space was never reclaimed.  A flush
or drain did not reclaim the space either and delete any log files.
>  
> Bryce Godfrey | Sr. Software Engineer | Azaleos Corporation
>  
> From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com] 
> Sent: Tuesday, May 22, 2012 1:10 PM
> To: user@cassandra.apache.org
> Subject: RE: 1.1 not removing commit log files?
>  
> The nodes appear to be holding steady at the 8G that I set it to in the config file now.
 I’ll keep an eye on them.
>  
> From: aaron morton [mailto:aaron@thelastpickle.com] 
> Sent: Tuesday, May 22, 2012 4:08 AM
> To: user@cassandra.apache.org
> Subject: Re: 1.1 not removing commit log files?
>  
> 4096 is also the internal hard coded default for commitlog_total_space_in_mb
>  
> If you are seeing more that 4GB of commit log files let us know. 
>  
> Cheers
>  
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>  
> On 22/05/2012, at 6:35 AM, Bryce Godfrey wrote:
>  
> 
> Thanks, I'll give it a try.
> 
> -----Original Message-----
> From: Alain RODRIGUEZ [mailto:arodrime@gmail.com] 
> Sent: Monday, May 21, 2012 2:12 AM
> To: user@cassandra.apache.org
> Subject: Re: 1.1 not removing commit log files?
> 
> commitlog_total_space_in_mb: 4096
> 
> By default this line is commented in 1.0.x if I remember well. I guess it is the same
in 1.1. You really should remove this comment or your commit logs will entirely fill up your
disk as it happened to me a while ago.
> 
> Alain
> 
> 2012/5/21 Pieter Callewaert <pieter.callewaert@be-mobile.be>:
> 
> Hi,
>  
>  
>  
> In 1.1 the commitlog files are pre-allocated with files of 128MB.
> (https://issues.apache.org/jira/browse/CASSANDRA-3411) This should
> however not exceed your commitlog size in Cassandra.yaml.
>  
>  
>  
> commitlog_total_space_in_mb: 4096
>  
>  
>  
> Kind regards,
>  
> Pieter Callewaert
>  
>  
>  
> From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
> Sent: maandag 21 mei 2012 9:52
> To: user@cassandra.apache.org
> Subject: 1.1 not removing commit log files?
>  
>  
>  
> The commit log drives on my nodes keep slowly filling up.  I don't see
> any errors in my logs that are indicating any issues that I can map to
> this issue.
>  
>  
>  
> Is this how 1.1 is supposed to work now?  Previous versions seemed to
> keep this drive at a minimum as it flushed.
>  
>  
>  
> /dev/mapper/mpathf     25G   21G  4.2G  83% /opt/cassandra/commitlog
>  
>  


Mime
View raw message