incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hiller, Dean" <Dean.Hil...@nrel.gov>
Subject Re: commitlog -deleted keyspaces.
Date Wed, 13 Mar 2013 12:34:09 GMT
I do not know if it is a known issue.  I only know what I saw in QA.  If I remember correctly,
I tested this on 1.1.4 and 1.2.2 with the same result that commit logs are left there(maybe
it is by design….I really don't know).  All I know is that the sstables do correctly grow
in size.

Dean

From: a k <kumaramit.extra@gmail.com<mailto:kumaramit.extra@gmail.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Tuesday, March 12, 2013 3:54 PM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: commitlog -deleted keyspaces.

Thanks Dean. I will try the node drain next, however Do you know if this is a known issue/bug
with 1.1, I scanned through some 200 odd jira entries that have commit log in the text for
some clues -but no luck.

Amit


On Tue, Mar 12, 2013 at 12:17 PM, Hiller, Dean <Dean.Hiller@nrel.gov<mailto:Dean.Hiller@nrel.gov>>
wrote:
Here is our cluster which has 10 billion rows on 6 nodes and about 1.2TB
[root@sdi-ci ~]# clush -g datanodes du -sh /opt/datastore/commitlog
a5: 1.1G /opt/datastore/commitlog
a3: 1.1G /opt/datastore/commitlog
a1: 1.1G /opt/datastore/commitlog
a2: 1006M /opt/datastore/commitlog
a4: 1.1G /opt/datastore/commitlog
a6: 1.1G /opt/datastore/commitlog

If you run nodetool drain on a node, you can wipe the commit logs after that(we QA tested
this but don't take my word for it).  We also found out drain was moving data to the sstables
but did not seem to delete from the commit log at all as commit log space used remained the
same after a drain.  We did not fully test removing the commit log files so you should try
to do that yourself with a test in QA.

Later,
Dean

From: a k <kumaramit.extra@gmail.com<mailto:kumaramit.extra@gmail.com><mailto:kumaramit.extra@gmail.com<mailto:kumaramit.extra@gmail.com>>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>"
<user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>
Date: Tuesday, March 12, 2013 10:46 AM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>"
<user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>
Subject: commitlog -deleted keyspaces.

We are running a 4 node cluster version 1.1.0 and our commit logs seem to be ever growing.
We have a total about 250 GB per node in the keyspaces/column families and the commit logs
are at about 30 GB.There have been several deletions of keyspaces in our setup and I am concerned
about a few things.

First being the size of the commit logs, we have not modified the  "commitlog_total_space_in_mb"
in yaml so I assume it is the default   4092MB, We don't have "memtable_flush_after_mins"
setting either.

Is this sane? Why would the size of the commit logs be so large (I am worried about the startup
time, when it has to replay the commit logs) and why would the default size of 4092 MB not
be enforced.

Would it cause us trouble when we upgrade to 1.2?

Another thing I have noticed is that upon restarts, the old keyspaces that were deleted re-appear
although with less data, I would imagine that has nothing to do with the commit logs.

Can I safely delete the commitlogs after the nodetool flush?



Many thanks
Amit



Mime
View raw message