incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From a k <kumaramit.ex...@gmail.com>
Subject Re: commitlog -deleted keyspaces.
Date Tue, 12 Mar 2013 22:54:32 GMT
Thanks Dean. I will try the node drain next, however Do you know if this is
a known issue/bug with 1.1, I scanned through some 200 odd jira entries
that have commit log in the text for some clues -but no luck.

Amit


On Tue, Mar 12, 2013 at 12:17 PM, Hiller, Dean <Dean.Hiller@nrel.gov> wrote:

> Here is our cluster which has 10 billion rows on 6 nodes and about 1.2TB
> [root@sdi-ci ~]# clush -g datanodes du -sh /opt/datastore/commitlog
> a5: 1.1G /opt/datastore/commitlog
> a3: 1.1G /opt/datastore/commitlog
> a1: 1.1G /opt/datastore/commitlog
> a2: 1006M /opt/datastore/commitlog
> a4: 1.1G /opt/datastore/commitlog
> a6: 1.1G /opt/datastore/commitlog
>
> If you run nodetool drain on a node, you can wipe the commit logs after
> that(we QA tested this but don't take my word for it).  We also found out
> drain was moving data to the sstables but did not seem to delete from the
> commit log at all as commit log space used remained the same after a drain.
>  We did not fully test removing the commit log files so you should try to
> do that yourself with a test in QA.
>
> Later,
> Dean
>
> From: a k <kumaramit.extra@gmail.com<mailto:kumaramit.extra@gmail.com>>
> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <
> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
> Date: Tuesday, March 12, 2013 10:46 AM
> To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <
> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
> Subject: commitlog -deleted keyspaces.
>
> We are running a 4 node cluster version 1.1.0 and our commit logs seem to
> be ever growing. We have a total about 250 GB per node in the
> keyspaces/column families and the commit logs are at about 30 GB.There have
> been several deletions of keyspaces in our setup and I am concerned about a
> few things.
>
> First being the size of the commit logs, we have not modified the
>  "commitlog_total_space_in_mb" in yaml so I assume it is the default
> 4092MB, We don't have "memtable_flush_after_mins" setting either.
>
> Is this sane? Why would the size of the commit logs be so large (I am
> worried about the startup time, when it has to replay the commit logs) and
> why would the default size of 4092 MB not be enforced.
>
> Would it cause us trouble when we upgrade to 1.2?
>
> Another thing I have noticed is that upon restarts, the old keyspaces that
> were deleted re-appear although with less data, I would imagine that has
> nothing to do with the commit logs.
>
> Can I safely delete the commitlogs after the nodetool flush?
>
>
>
> Many thanks
> Amit
>
>

Mime
View raw message