incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Donald Smith <Donald.Sm...@audiencescience.com>
Subject RE: Lots of commitlog files
Date Mon, 14 Apr 2014 19:50:24 GMT
Another thing.   cassandra.yaml says:

# Total space to use for commitlogs.  Since commitlog segments are
# mmapped, and hence use up address space, the default size is 32
# on 32-bit JVMs, and 1024 on 64-bit JVMs.
#
# If space gets above this value (it will round up to the next nearest
# segment multiple), Cassandra will flush every dirty CF in the oldest
# segment and remove it.  So a small total commitlog space will tend
# to cause more flush activity on less-active columnfamilies.
# commitlog_total_space_in_mb: 4096

We're using a 64 bit linux with a 64 bit JVM:

Java(TM) SE Runtime Environment (build 1.7.0_40-b43)
Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, mixed mode)

but our commit log files are each 32MB in size. Is this indicative of a bug?  Shouldn't they
be 1024MB in size?

  Don

From: Donald Smith
Sent: Monday, April 14, 2014 12:04 PM
To: 'user@cassandra.apache.org'
Subject: Logs of commitlog files

1. With cassandra 2.0.6, we have 547G of files in /var/lib/commitlog/.  I started a "nodetool
flush" 65 minutes ago; it's still running.  The 17536 commitlog files have been created in
the last 3 days.  (The node has 2.1T of sstables data in /var/lib/cassandra/data/.  This is
in staging, not prod.) Why so many commit logs?  Here are our commitlog-related settings in
cassandra.yaml:
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
# The size of the individual commitlog file segments.  A commitlog
# archiving commitlog segments (see commitlog_archiving.properties),
commitlog_segment_size_in_mb: 32
# Total space to use for commitlogs.  Since commitlog segments are
# segment and remove it.  So a small total commitlog space will tend
# commitlog_total_space_in_mb: 4096

Maybe we should set commitlog_total_space_in_mb to something other than the default. According
to OpsCenter, commitlog_total_space_in_mb is "None".    But it seems odd that there'd be so
many commit logs.

The node is under heavy write load.   There are about 2900 compactions pending.

We are NOT archiving commitlogs, via commitlog_archiving.properties.

BTW, the documentation for nodetool<http://wiki.apache.org/cassandra/NodeTool> says:
Flush

Flushes memtables (in memory) to SSTables (on disk), which also enables CommitLog<http://wiki.apache.org/cassandra/CommitLog>
segments to be deleted.
But even after doing a flush, the /var/lib/commitlog dir still has 1G of files, even after
waiting 30  minutes.  Each file is 32M in size, plus or minus a few bytes.  I tried this on
other clusters, with much smaller amounts of data.   Even restarting Cassandra doesn't help.

I surmise that the 1GB of commit logs are normal: they probably allocate that space as a workspace.


Thanks,  Don

Donald A. Smith | Senior Software Engineer
P: 425.201.3900 x 3866
C: (206) 819-5965
F: (646) 443-2333
donalds@AudienceScience.com<mailto:donalds@AudienceScience.com>

[AudienceScience]


Mime
View raw message