So this happened to me again, but it was only when the cluster had a node down for a while.  Then the commit logs started piling up past the limit I set in the config file, and filled the drive. 

After the node recovered and hints had replayed the space was never reclaimed.  A flush or drain did not reclaim the space either and delete any log files.

 

Bryce Godfrey | Sr. Software Engineer | Azaleos Corporation

 

From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
Sent: Tuesday, May 22, 2012 1:10 PM
To: user@cassandra.apache.org
Subject: RE: 1.1 not removing commit log files?

 

The nodes appear to be holding steady at the 8G that I set it to in the config file now.  I’ll keep an eye on them.

 

From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Tuesday, May 22, 2012 4:08 AM
To: user@cassandra.apache.org
Subject: Re: 1.1 not removing commit log files?

 

4096 is also the internal hard coded default for commitlog_total_space_in_mb

 

If you are seeing more that 4GB of commit log files let us know. 

 

Cheers

 

-----------------

Aaron Morton

Freelance Developer

@aaronmorton

 

On 22/05/2012, at 6:35 AM, Bryce Godfrey wrote:

 

Thanks, I'll give it a try.

-----Original Message-----
From: Alain RODRIGUEZ [mailto:arodrime@gmail.com]
Sent: Monday, May 21, 2012 2:12 AM
To: user@cassandra.apache.org
Subject: Re: 1.1 not removing commit log files?

commitlog_total_space_in_mb: 4096

By default this line is commented in 1.0.x if I remember well. I guess it is the same in 1.1. You really should remove this comment or your commit logs will entirely fill up your disk as it happened to me a while ago.

Alain

2012/5/21 Pieter Callewaert <pieter.callewaert@be-mobile.be>:

Hi,

 

 

 

In 1.1 the commitlog files are pre-allocated with files of 128MB.

(https://issues.apache.org/jira/browse/CASSANDRA-3411) This should

however not exceed your commitlog size in Cassandra.yaml.

 

 

 

commitlog_total_space_in_mb: 4096

 

 

 

Kind regards,

 

Pieter Callewaert

 

 

 

From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]

Sent: maandag 21 mei 2012 9:52

To: user@cassandra.apache.org

Subject: 1.1 not removing commit log files?

 

 

 

The commit log drives on my nodes keep slowly filling up.  I don't see

any errors in my logs that are indicating any issues that I can map to

this issue.

 

 

 

Is this how 1.1 is supposed to work now?  Previous versions seemed to

keep this drive at a minimum as it flushed.

 

 

 

/dev/mapper/mpathf     25G   21G  4.2G  83% /opt/cassandra/commitlog