cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From a k <>
Subject Re: commitlog -deleted keyspaces.
Date Tue, 12 Mar 2013 22:54:32 GMT
Thanks Dean. I will try the node drain next, however Do you know if this is
a known issue/bug with 1.1, I scanned through some 200 odd jira entries
that have commit log in the text for some clues -but no luck.


On Tue, Mar 12, 2013 at 12:17 PM, Hiller, Dean <> wrote:

> Here is our cluster which has 10 billion rows on 6 nodes and about 1.2TB
> [root@sdi-ci ~]# clush -g datanodes du -sh /opt/datastore/commitlog
> a5: 1.1G /opt/datastore/commitlog
> a3: 1.1G /opt/datastore/commitlog
> a1: 1.1G /opt/datastore/commitlog
> a2: 1006M /opt/datastore/commitlog
> a4: 1.1G /opt/datastore/commitlog
> a6: 1.1G /opt/datastore/commitlog
> If you run nodetool drain on a node, you can wipe the commit logs after
> that(we QA tested this but don't take my word for it).  We also found out
> drain was moving data to the sstables but did not seem to delete from the
> commit log at all as commit log space used remained the same after a drain.
>  We did not fully test removing the commit log files so you should try to
> do that yourself with a test in QA.
> Later,
> Dean
> From: a k <<>>
> Reply-To: "<>" <
> Date: Tuesday, March 12, 2013 10:46 AM
> To: "<>" <
> Subject: commitlog -deleted keyspaces.
> We are running a 4 node cluster version 1.1.0 and our commit logs seem to
> be ever growing. We have a total about 250 GB per node in the
> keyspaces/column families and the commit logs are at about 30 GB.There have
> been several deletions of keyspaces in our setup and I am concerned about a
> few things.
> First being the size of the commit logs, we have not modified the
>  "commitlog_total_space_in_mb" in yaml so I assume it is the default
> 4092MB, We don't have "memtable_flush_after_mins" setting either.
> Is this sane? Why would the size of the commit logs be so large (I am
> worried about the startup time, when it has to replay the commit logs) and
> why would the default size of 4092 MB not be enforced.
> Would it cause us trouble when we upgrade to 1.2?
> Another thing I have noticed is that upon restarts, the old keyspaces that
> were deleted re-appear although with less data, I would imagine that has
> nothing to do with the commit logs.
> Can I safely delete the commitlogs after the nodetool flush?
> Many thanks
> Amit

View raw message