incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kasun Weranga <kas...@wso2.com>
Subject Re: Cassandra commitlog directory size increase on every restart - Cassandra 1.1.0
Date Thu, 09 Aug 2012 14:15:02 GMT
Any idea on how to fix this?

Thanks,
Kasun

On Wed, Aug 8, 2012 at 11:56 AM, Kasun Weranga <kasunw@wso2.com> wrote:

> Hi all,
>
> I am facing the above issue in Cassandra 1.1.0, it will add 134.2MB
> commitlog file in every restart, but it never delete it. We can't control
> the commitlog dir size even
> by explicitly setting commitlog_total_space_in_mb in  cassandra.yaml.
> I set commitlog_total_space_in_mb as 512 in cassandra.yaml and did some
> testing, now my commitlog directory size reached 1.1 GB.
>
> Also I turned on debug log for CommitLog, This is what I get when server
> starts.
>
>
> [2012-08-08 11:25:30,860]  INFO
> {org.apache.cassandra.config.DatabaseDescriptor} -  Global memtable
> threshold is enabled at 151MB
> [2012-08-08 11:25:31,427]  INFO
> {org.apache.cassandra.service.CacheService} -  Initializing key cache with
> capacity of 12 MBs.
> [2012-08-08 11:25:31,443]  INFO
> {org.apache.cassandra.service.CacheService} -  Scheduling key cache save to
> each 14400 seconds (going to save all keys).
> [2012-08-08 11:25:31,445]  INFO
> {org.apache.cassandra.service.CacheService} -  Initializing row cache with
> capacity of 0 MBs and provider
> org.apache.cassandra.cache.SerializingCacheProvider
> [2012-08-08 11:25:31,448]  INFO
> {org.apache.cassandra.service.CacheService} -  Scheduling row cache save to
> each 0 seconds (going to save all keys).
> [2012-08-08 11:25:31,677]  INFO
> {org.apache.cassandra.io.sstable.SSTableReader} -  Opening
> ./repository/database/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-6
> (1644 bytes)
> [2012-08-08 11:25:31,677]  INFO
> {org.apache.cassandra.io.sstable.SSTableReader} -  Opening
> ./repository/database/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-5
> (5967 bytes)
> [2012-08-08 11:25:31,784]  INFO
> {org.apache.cassandra.io.sstable.SSTableReader} -  Opening
> ./repository/database/cassandra/data/system/Versions/system-Versions-hc-5
> (247 bytes)
> [2012-08-08 11:25:31,789]  INFO
> {org.apache.cassandra.io.sstable.SSTableReader} -  Opening
> ./repository/database/cassandra/data/system/Versions/system-Versions-hc-6
> (247 bytes)
> [2012-08-08 11:25:31,887]  INFO
> {org.apache.cassandra.io.sstable.SSTableReader} -  Opening
> ./repository/database/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hc-1
> (242 bytes)
> [2012-08-08 11:25:31,897]  INFO
> {org.apache.cassandra.io.sstable.SSTableReader} -  Opening
> ./repository/database/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hc-2
> (244 bytes)
> [2012-08-08 11:25:31,935]  INFO
> {org.apache.cassandra.io.sstable.SSTableReader} -  Opening
> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-14
> (80 bytes)
> [2012-08-08 11:25:31,935]  INFO
> {org.apache.cassandra.io.sstable.SSTableReader} -  Opening
> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-13
> (346 bytes)
> [2012-08-08 11:25:31,935]  INFO
> {org.apache.cassandra.io.sstable.SSTableReader} -  Opening
> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-15
> (163 bytes)
> [2012-08-08 11:25:32,980]  INFO
> {org.apache.cassandra.io.sstable.SSTableReader} -  Opening
> ./repository/database/cassandra/data/EVENT_KS/org_wso2_sample_httpd_logs/EVENT_KS-org_wso2_sample_httpd_logs-hc-2
> (312328 bytes)
> [2012-08-08 11:25:32,980]  INFO
> {org.apache.cassandra.io.sstable.SSTableReader} -  Opening
> ./repository/database/cassandra/data/EVENT_KS/org_wso2_sample_httpd_logs/EVENT_KS-org_wso2_sample_httpd_logs-hc-3
> (312992 bytes)
> [2012-08-08 11:25:32,984]  INFO
> {org.apache.cassandra.io.sstable.SSTableReader} -  Opening
> ./repository/database/cassandra/data/EVENT_KS/org_wso2_sample_httpd_logs/EVENT_KS-org_wso2_sample_httpd_logs-hc-1
> (647552 bytes)
> [2012-08-08 11:25:33,183]  INFO
> {org.apache.cassandra.io.sstable.SSTableReader} -  Opening
> ./repository/database/cassandra/data/EVENT_KS/bam_service_data_publisher/EVENT_KS-bam_service_data_publisher-hc-1
> (249143 bytes)
> [2012-08-08 11:25:33,266]  INFO
> {org.apache.cassandra.io.sstable.SSTableReader} -  Opening
> ./repository/database/cassandra/data/META_KS/STREAM_DEFINITION_ID_TO_KEY/META_KS-STREAM_DEFINITION_ID_TO_KEY-hc-1
> (152 bytes)
> [2012-08-08 11:25:33,266]  INFO
> {org.apache.cassandra.io.sstable.SSTableReader} -  Opening
> ./repository/database/cassandra/data/META_KS/STREAM_DEFINITION_ID_TO_KEY/META_KS-STREAM_DEFINITION_ID_TO_KEY-hc-2
> (150 bytes)
> [2012-08-08 11:25:33,362]  INFO
> {org.apache.cassandra.io.sstable.SSTableReader} -  Opening
> ./repository/database/cassandra/data/META_KS/STREAM_DEFINITION/META_KS-STREAM_DEFINITION-hc-2
> (319 bytes)
> [2012-08-08 11:25:33,363]  INFO
> {org.apache.cassandra.io.sstable.SSTableReader} -  Opening
> ./repository/database/cassandra/data/META_KS/STREAM_DEFINITION/META_KS-STREAM_DEFINITION-hc-1
> (484 bytes)
> [2012-08-08 11:25:33,412]  INFO
> {org.apache.cassandra.io.sstable.SSTableReader} -  Opening
> ./repository/database/cassandra/data/META_KS/STREAM_ID/META_KS-STREAM_ID-hc-1
> (163 bytes)
> [2012-08-08 11:25:33,422]  INFO
> {org.apache.cassandra.io.sstable.SSTableReader} -  Opening
> ./repository/database/cassandra/data/META_KS/STREAM_ID/META_KS-STREAM_ID-hc-2
> (161 bytes)
> [2012-08-08 11:25:33,447]  INFO
> {org.apache.cassandra.service.AbstractCassandraDaemon} -  completed
> pre-loading (6 keys) key cache.
> [2012-08-08 11:25:34,227]  INFO
> {org.apache.cassandra.db.commitlog.CommitLog} -  Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809612463142.log,
> ./repository/database/cassandra/commitlog/CommitLog-15809986125630.log,
> ./repository/database/cassandra/commitlog/CommitLog-15809986875007.log,
> ./repository/database/cassandra/commitlog/CommitLog-15809986654744.log,
> ./repository/database/cassandra/commitlog/CommitLog-15809986776035.log,
> ./repository/database/cassandra/commitlog/CommitLog-15809986520160.log,
> ./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
> [2012-08-08 11:25:34,230]  INFO
> {org.apache.cassandra.db.commitlog.CommitLog} -  Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809612463142.log
> [2012-08-08 11:25:34,255] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809612463142.log
> starting at 0
> [2012-08-08 11:25:34,255] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Reading mutation at 0
> [2012-08-08 11:25:34,295] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  replaying mutation for
> system.6275696c64: {ColumnFamily(Versions [version:false:5@1344363219522000
> ,])}
> [2012-08-08 11:25:34,322] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Reading mutation at 91
> [2012-08-08 11:25:34,322] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  replaying mutation for
> system.63716c: {ColumnFamily(Versions [version:false:5@1344363219527000
> ,])}
> [2012-08-08 11:25:34,322] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Reading mutation at 180
> [2012-08-08 11:25:34,323] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  replaying mutation for
> system.746872696674: {ColumnFamily(Versions
> [version:false:7@1344363219528000,])}
> [2012-08-08 11:25:34,323] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Reading mutation at 274
> [2012-08-08 11:25:34,324] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  replaying mutation for
> system.4c: {ColumnFamily(LocationInfo
> [47656e65726174696f6e:false:4@1344363219567000,])}
> [2012-08-08 11:25:34,325] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Reading mutation at 363
> [2012-08-08 11:25:34,325] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  replaying mutation for
> system.426f6f747374726170: {ColumnFamily(LocationInfo
> [42:false:1@1344363219785000,])}
> [2012-08-08 11:25:34,325] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Reading mutation at 448
> [2012-08-08 11:25:34,325] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  replaying mutation for
> system.4c: {ColumnFamily(LocationInfo [546f6b656e:false:16@1344363219786000
> ,])}
> [2012-08-08 11:25:34,326] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Reading mutation at 544
> [2012-08-08 11:25:34,326] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Encountered end of segment
> marker at 548
> [2012-08-08 11:25:34,326]  INFO
> {org.apache.cassandra.db.commitlog.CommitLog} -  Finished reading
> ./repository/database/cassandra/commitlog/CommitLog-15809612463142.log
> [2012-08-08 11:25:34,326]  INFO
> {org.apache.cassandra.db.commitlog.CommitLog} -  Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986125630.log
> [2012-08-08 11:25:34,359] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986125630.log
> starting at 0
> [2012-08-08 11:25:34,359] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Reading mutation at 0
> [2012-08-08 11:25:34,359] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Encountered end of segment
> marker at 4
> [2012-08-08 11:25:34,360]  INFO
> {org.apache.cassandra.db.commitlog.CommitLog} -  Finished reading
> ./repository/database/cassandra/commitlog/CommitLog-15809986125630.log
> [2012-08-08 11:25:34,360]  INFO
> {org.apache.cassandra.db.commitlog.CommitLog} -  Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986875007.log
> [2012-08-08 11:25:34,392] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986875007.log
> starting at 0
> [2012-08-08 11:25:34,392] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Reading mutation at 0
> [2012-08-08 11:25:34,393] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Encountered end of segment
> marker at 4
> [2012-08-08 11:25:34,393]  INFO
> {org.apache.cassandra.db.commitlog.CommitLog} -  Finished reading
> ./repository/database/cassandra/commitlog/CommitLog-15809986875007.log
> [2012-08-08 11:25:34,393]  INFO
> {org.apache.cassandra.db.commitlog.CommitLog} -  Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986654744.log
> [2012-08-08 11:25:34,429] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986654744.log
> starting at 0
> [2012-08-08 11:25:34,429] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Reading mutation at 0
> [2012-08-08 11:25:34,429] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Encountered end of segment
> marker at 4
> [2012-08-08 11:25:34,430]  INFO
> {org.apache.cassandra.db.commitlog.CommitLog} -  Finished reading
> ./repository/database/cassandra/commitlog/CommitLog-15809986654744.log
> [2012-08-08 11:25:34,430]  INFO
> {org.apache.cassandra.db.commitlog.CommitLog} -  Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986776035.log
> [2012-08-08 11:25:34,449] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986776035.log
> starting at 0
> [2012-08-08 11:25:34,450] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Reading mutation at 0
> [2012-08-08 11:25:34,450] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Encountered end of segment
> marker at 4
> [2012-08-08 11:25:34,451]  INFO
> {org.apache.cassandra.db.commitlog.CommitLog} -  Finished reading
> ./repository/database/cassandra/commitlog/CommitLog-15809986776035.log
> [2012-08-08 11:25:34,451]  INFO
> {org.apache.cassandra.db.commitlog.CommitLog} -  Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986520160.log
> [2012-08-08 11:25:34,493] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986520160.log
> starting at 0
> [2012-08-08 11:25:34,493] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Reading mutation at 0
> [2012-08-08 11:25:34,494] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Encountered end of segment
> marker at 4
> [2012-08-08 11:25:34,494]  INFO
> {org.apache.cassandra.db.commitlog.CommitLog} -  Finished reading
> ./repository/database/cassandra/commitlog/CommitLog-15809986520160.log
> [2012-08-08 11:25:34,494]  INFO
> {org.apache.cassandra.db.commitlog.CommitLog} -  Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
> [2012-08-08 11:25:34,515] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
> starting at 0
> [2012-08-08 11:25:34,515] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Reading mutation at 0
> [2012-08-08 11:25:34,515] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Encountered end of segment
> marker at 4
> [2012-08-08 11:25:34,516]  INFO
> {org.apache.cassandra.db.commitlog.CommitLog} -  Finished reading
> ./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
> [2012-08-08 11:25:34,516] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  Finished waiting on
> mutations from recovery
> [2012-08-08 11:25:34,519]  INFO
> {org.apache.cassandra.db.ColumnFamilyStore} -  Enqueuing flush of
> Memtable-Versions@507924927(83/103 serialized/live bytes, 3 ops)
> [2012-08-08 11:25:34,520]  INFO {org.apache.cassandra.db.Memtable} -
>  Writing Memtable-Versions@507924927(83/103 serialized/live bytes, 3 ops)
> [2012-08-08 11:25:34,840]  INFO {org.apache.cassandra.db.Memtable} -
>  Completed flushing
> ./repository/database/cassandra/data/system/Versions/system-Versions-hc-7-Data.db
> (247 bytes)
> [2012-08-08 11:25:34,858] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  discard completed log
> segments for ReplayPosition(segmentId=2233031438605, position=0), column
> family 7
> [2012-08-08 11:25:34,859] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  *Not safe to delete
> commit log
> CommitLogSegment(./repository/database/cassandra/commitlog/CommitLog-2233031438605.log);
> dirty is ; hasNext: false*
> [2012-08-08 11:25:34,859]  INFO
> {org.apache.cassandra.db.commitlog.CommitLog} -  Log replay complete, 3
> replayed mutations
> [2012-08-08 11:25:34,879]  INFO
> {org.apache.cassandra.service.StorageService} -  Cassandra version: 1.1.0
> [2012-08-08 11:25:34,879]  INFO
> {org.apache.cassandra.service.StorageService} -  Thrift API version: 19.30.0
> [2012-08-08 11:25:34,881]  INFO
> {org.apache.cassandra.service.StorageService} -  CQL supported versions:
> 2.0.0,3.0.0-beta1 (default: 2.0.0)
> [2012-08-08 11:25:34,904]  INFO
> {org.apache.cassandra.service.StorageService} -  Loading persisted ring
> state
> [2012-08-08 11:25:34,906]  INFO
> {org.apache.cassandra.service.StorageService} -  Starting up server gossip
> [2012-08-08 11:25:34,917]  INFO
> {org.apache.cassandra.db.ColumnFamilyStore} -  Enqueuing flush of
> Memtable-LocationInfo@1885256814(29/36 serialized/live bytes, 1 ops)
> [2012-08-08 11:25:34,918]  INFO {org.apache.cassandra.db.Memtable} -
>  Writing Memtable-LocationInfo@1885256814(29/36 serialized/live bytes, 1
> ops)
> [2012-08-08 11:25:35,182]  INFO {org.apache.cassandra.db.Memtable} -
>  Completed flushing
> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-16-Data.db
> (80 bytes)
> [2012-08-08 11:25:35,183] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  discard completed log
> segments for ReplayPosition(segmentId=2233031438605, position=363), column
> family 0
> [2012-08-08 11:25:35,184] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - * Not safe to delete
> commit log
> CommitLogSegment(./repository/database/cassandra/commitlog/CommitLog-2233031438605.log);
> dirty is Versions (7), ; hasNext: false*
> [2012-08-08 11:25:35,192]  INFO
> {org.apache.cassandra.db.compaction.CompactionTask} -  Compacting
> [SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-13-Data.db'),
> SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-14-Data.db'),
> SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-16-Data.db'),
> SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-15-Data.db')]
> [2012-08-08 11:25:35,390]  INFO
> {org.apache.cassandra.net.MessagingService} -  Starting Messaging Service
> on port 7000
> [2012-08-08 11:25:35,399]  INFO
> {org.apache.cassandra.service.StorageService} -  Using saved token
> 83957406514855090039949046883112789193
> [2012-08-08 11:25:35,401]  INFO
> {org.apache.cassandra.db.ColumnFamilyStore} -  Enqueuing flush of
> Memtable-LocationInfo@1345718067(53/66 serialized/live bytes, 2 ops)
> [2012-08-08 11:25:35,401]  INFO {org.apache.cassandra.db.Memtable} -
>  Writing Memtable-LocationInfo@1345718067(53/66 serialized/live bytes, 2
> ops)
> [2012-08-08 11:25:35,959]  INFO
> {org.apache.cassandra.db.compaction.CompactionTask} -  Compacted to
> [./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-17-Data.db,].
>  669 to 346 (~51% of original) bytes for 3 keys at 0.000502MB/s.  Time:
> 657ms.
> [2012-08-08 11:25:36,001]  INFO {org.apache.cassandra.db.Memtable} -
>  Completed flushing
> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-18-Data.db
> (163 bytes)
> [2012-08-08 11:25:36,002] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  discard completed log
> segments for ReplayPosition(segmentId=2233031438605, position=544), column
> family 0
> [2012-08-08 11:25:36,002] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} -  *Not safe to delete
> commit log
> CommitLogSegment(./repository/database/cassandra/commitlog/CommitLog-2233031438605.log);
> dirty is Versions (7), ; hasNext: false*
> [2012-08-08 11:25:36,003]  INFO
> {org.apache.cassandra.service.StorageService} -  Node localhost/127.0.0.1state jump to
normal
> [2012-08-08 11:25:36,009]  INFO
> {org.apache.cassandra.service.StorageService} -  Bootstrap/Replace/Move
> completed! Now serving reads.
> [2012-08-08 11:25:36,010]  INFO {org.apache.cassandra.utils.Mx4jTool} -
>  Will not load MX4J, mx4j-tools.jar is not in the classpath
> [2012-08-08 11:25:36,058]  INFO
> {org.apache.axis2.deployment.ClusterBuilder} -  Clustering has been disabled
> [2012-08-08 11:25:36,173]  INFO
> {org.apache.cassandra.thrift.CassandraDaemon} -  Binding thrift service to
> localhost/127.0.0.1:9160
> [2012-08-08 11:25:36,254]  INFO
> {org.apache.cassandra.thrift.CassandraDaemon} -  Using TFastFramedTransport
> with a max frame size of 15728640 bytes.
> [2012-08-08 11:25:36,257]  INFO
> {org.apache.cassandra.thrift.CassandraDaemon} -  Using
> synchronous/threadpool thrift server on localhost/127.0.0.1 : 9160
> [2012-08-08 11:25:36,258]  INFO
> {org.wso2.carbon.cassandra.server.CassandraServerController} -  Cassandra
> Server Controller Thread was destroyed successfully
> [2012-08-08 11:25:36,258]  INFO
> {org.apache.cassandra.thrift.CassandraDaemon} -  Listening for thrift
> clients...
>
>
> What is the reason for keeping old commitlogs eventhough I have
> set commitlog_total_space_in_mb in cassandra.yaml?
> Why is this printing "Not safe to delete" message?
>
>
> Appreciate your help on fixing this issue.
>
>
> Thanks,
> Kasun.
>

Mime
View raw message