So this happened to me again, but it was only when the cluster had a node down for a while.
Then the commit logs started piling up past the limit I set in the config file, and filled
the drive.
After the node recovered and hints had replayed the space was never reclaimed. A flush or
drain did not reclaim the space either and delete any log files.
Bryce Godfrey | Sr. Software Engineer | Azaleos Corporation<http://www.azaleos.com/>
From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
Sent: Tuesday, May 22, 2012 1:10 PM
To: user@cassandra.apache.org
Subject: RE: 1.1 not removing commit log files?
The nodes appear to be holding steady at the 8G that I set it to in the config file now.
I'll keep an eye on them.
From: aaron morton [mailto:aaron@thelastpickle.com]<mailto:[mailto:aaron@thelastpickle.com]>
Sent: Tuesday, May 22, 2012 4:08 AM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: 1.1 not removing commit log files?
4096 is also the internal hard coded default for commitlog_total_space_in_mb
If you are seeing more that 4GB of commit log files let us know.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 22/05/2012, at 6:35 AM, Bryce Godfrey wrote:
Thanks, I'll give it a try.
-----Original Message-----
From: Alain RODRIGUEZ [mailto:arodrime@gmail.com]<mailto:[mailto:arodrime@gmail.com]>
Sent: Monday, May 21, 2012 2:12 AM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: 1.1 not removing commit log files?
commitlog_total_space_in_mb: 4096
By default this line is commented in 1.0.x if I remember well. I guess it is the same in 1.1.
You really should remove this comment or your commit logs will entirely fill up your disk
as it happened to me a while ago.
Alain
2012/5/21 Pieter Callewaert <pieter.callewaert@be-mobile.be<mailto:pieter.callewaert@be-mobile.be>>:
Hi,
In 1.1 the commitlog files are pre-allocated with files of 128MB.
(https://issues.apache.org/jira/browse/CASSANDRA-3411) This should
however not exceed your commitlog size in Cassandra.yaml.
commitlog_total_space_in_mb: 4096
Kind regards,
Pieter Callewaert
From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]<mailto:[mailto:Bryce.Godfrey@azaleos.com]>
Sent: maandag 21 mei 2012 9:52
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: 1.1 not removing commit log files?
The commit log drives on my nodes keep slowly filling up. I don't see
any errors in my logs that are indicating any issues that I can map to
this issue.
Is this how 1.1 is supposed to work now? Previous versions seemed to
keep this drive at a minimum as it flushed.
/dev/mapper/mpathf 25G 21G 4.2G 83% /opt/cassandra/commitlog
|