cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hiller, Dean" <>
Subject Re: commitlog -deleted keyspaces.
Date Tue, 12 Mar 2013 18:17:09 GMT
Here is our cluster which has 10 billion rows on 6 nodes and about 1.2TB
[root@sdi-ci ~]# clush -g datanodes du -sh /opt/datastore/commitlog
a5: 1.1G /opt/datastore/commitlog
a3: 1.1G /opt/datastore/commitlog
a1: 1.1G /opt/datastore/commitlog
a2: 1006M /opt/datastore/commitlog
a4: 1.1G /opt/datastore/commitlog
a6: 1.1G /opt/datastore/commitlog

If you run nodetool drain on a node, you can wipe the commit logs after that(we QA tested
this but don't take my word for it).  We also found out drain was moving data to the sstables
but did not seem to delete from the commit log at all as commit log space used remained the
same after a drain.  We did not fully test removing the commit log files so you should try
to do that yourself with a test in QA.


From: a k <<>>
Reply-To: "<>" <<>>
Date: Tuesday, March 12, 2013 10:46 AM
To: "<>" <<>>
Subject: commitlog -deleted keyspaces.

We are running a 4 node cluster version 1.1.0 and our commit logs seem to be ever growing.
We have a total about 250 GB per node in the keyspaces/column families and the commit logs
are at about 30 GB.There have been several deletions of keyspaces in our setup and I am concerned
about a few things.

First being the size of the commit logs, we have not modified the  "commitlog_total_space_in_mb"
in yaml so I assume it is the default   4092MB, We don't have "memtable_flush_after_mins"
setting either.

Is this sane? Why would the size of the commit logs be so large (I am worried about the startup
time, when it has to replay the commit logs) and why would the default size of 4092 MB not
be enforced.

Would it cause us trouble when we upgrade to 1.2?

Another thing I have noticed is that upon restarts, the old keyspaces that were deleted re-appear
although with less data, I would imagine that has nothing to do with the commit logs.

Can I safely delete the commitlogs after the nodetool flush?

Many thanks

View raw message