incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julie <julie.su...@nextcentury.com>
Subject Re: Cassandra disk space utilization WAY higher than I would expect
Date Thu, 19 Aug 2010 14:23:58 GMT
Peter Schuller <peter.schuller <at> infidyne.com> writes:

> Without necessarily dumping all the information - approximately what
> do they contain? Do they contain anything about compactions,
> anti-compactions, streaming, etc?
> 
> With an idle node after taking writes, I *think* the only expected
> disk I/O (once it has settled) would be a memtable flush triggered by
> memtable_flush_after_mins, and possibly compactions resulting from
> that (depending on how close one were to triggering compaction prior
> to the memtable flush). Whatever is causing additional sstables to be
> written, even if somehow triggered incorrectly, I'd hope that they
> were logged still.

******************************************************
Hi Peter,

Thanks for your ideas.  Here is what I see in the bloated node's system.log file
(condensed) starting 4 minutes after the last write.  At this time, data usage
on this node was only around 28GB (as expected).
******************************************************

INFO [COMPACTION-POOL:1] 2010-08-16 13:51:38,432 CompactionManager.java
 (line 246) Compacting ...
 INFO [COMPACTION-POOL:1] 2010-08-16 14:20:02,840 CompactionManager.java
 (line 246) Compacting ...
 INFO [COMPACTION-POOL:1] 2010-08-16 14:28:37,085 CompactionManager.java
 (line 246) Compacting ...

 INFO [Thread-25] 2010-08-16 14:34:06,827 StreamCompletionHandler.java
 (line 64) Streaming added
 /var/lib/cassandra/data/Keyspace1/Standard1-467-Data.db

 INFO [Timer-1] 2010-08-16 14:34:31,655 Gossiper.java (line 179)
 InetAddress /10.254.107.178 is now dead.
 INFO [GMFD:1] 2010-08-16 14:35:03,012 Gossiper.java (line 568)
 InetAddress /10.254.107.178 is now UP

 INFO [COMPACTION-POOL:1] 2010-08-16 14:36:04,808 CompactionManager.java
 (line 246) Compacting ...
 INFO [COMPACTION-POOL:1] 2010-08-16 14:40:31,283 CompactionManager.java
 (line 246) Compacting ...
 INFO [FLUSH-TIMER] 2010-08-16 14:50:46,570 ColumnFamilyStore.java (line 357)
 Standard1 has reached its threshold; switching in a fresh Memtable at
 CommitLogContext(file=
 '/var/lib/cassandra/commitlog/CommitLog-1281980828674.log', position=87493038)


 INFO [FLUSH-TIMER] 2010-08-16 14:50:46,602 ColumnFamilyStore.java (line 609)
 Enqueuing flush of Memtable(Standard1)@31873581
 INFO [FLUSH-WRITER-POOL:1] 2010-08-16 14:50:46,647 Memtable.java (line 148)
 Writing Memtable(Standard1)@31873581
 INFO [GC inspection] 2010-08-16 14:50:47,350 GCInspector.java (line 110)
 GC for ConcurrentMarkSweep: 362 ms, 506927464 reclaimed leaving 129128920 used;
 max is 1172766720
 INFO [FLUSH-WRITER-POOL:1] 2010-08-16 14:50:51,137 Memtable.java (line 162)
 Completed flushing /var/lib/cassandra/data/Keyspace1/Standard1-472-Data.db
 INFO [SSTABLE-CLEANUP-TIMER] 2010-08-16 14:50:58,411
 SSTableDeletingReference.java (line 104)
 Deleted /var/lib/cassandra/data/Keyspace1/Standard1-223-Data.db
...
...
 INFO [SSTABLE-CLEANUP-TIMER] 2010-08-16 14:51:40,662
 SSTableDeletingReference.java (line 104)
 Deleted /var/lib/cassandra/data/Keyspace1/Standard1-407-Data.db

 INFO [Timer-1] 2010-08-16 14:51:40,772 Gossiper.java (line 179)
 InetAddress /10.254.242.159 is now dead.

 INFO [SSTABLE-CLEANUP-TIMER] 2010-08-16 14:51:41,011
 SSTableDeletingReference.java (line 104)
 Deleted /var/lib/cassandra/data/Keyspace1/Standard1-279-Data.db
 ...
 ...
 INFO [SSTABLE-CLEANUP-TIMER] 2010-08-16 14:52:22,206
 SSTableDeletingReference.java (line 104)
 Deleted /var/lib/cassandra/data/Keyspace1/Standard1-389-Data.db

 INFO [GMFD:1] 2010-08-16 14:52:34,906 Gossiper.java (line 568)
 InetAddress /10.254.242.159 is now UP

 INFO [COMPACTION-POOL:1] 2010-08-16 14:58:48,958 CompactionManager.java
 (line 246) Compacting ...
 INFO [COMPACTION-POOL:1] 2010-08-16 15:02:43,444 CompactionManager.java
 (line 246) Compacting ...
 INFO [COMPACTION-POOL:1] 2010-08-16 15:05:08,837 CompactionManager.java
 (line 246) Compacting []

 INFO [Timer-1] 2010-08-16 15:17:38,697 Gossiper.java (line 179)
 InetAddress /10.254.234.226 is now dead.
 INFO [GMFD:1] 2010-08-16 15:18:37,471 Gossiper.java (line 568)
 InetAddress /10.254.234.226 is now UP


 INFO [Thread-28] 2010-08-16 15:38:34,685 StreamCompletionHandler.java
 (line 64)
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-475-Data.db

 INFO [COMPACTION-POOL:1] 2010-08-16 16:05:08,811 CompactionManager.java
 (line 246) Compacting []
 INFO [COMPACTION-POOL:1] 2010-08-16 17:05:08,792 CompactionManager.java
 (line 246) Compacting []

 INFO [Timer-1] 2010-08-16 17:05:55,583 Gossiper.java (line 179)
 InetAddress /10.254.242.159 is now dead.
 INFO [GMFD:1] 2010-08-16 17:06:19,614 Gossiper.java (line 568)
 InetAddress /10.254.242.159 is now UP


 INFO [Thread-34] 2010-08-16 17:24:57,584 StreamCompletionHandler.java
 (line 64)
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-477-Data.db

 INFO [Timer-1] 2010-08-16 17:25:19,206 Gossiper.java (line 179)
 InetAddress /10.210.157.187 is now dead.
 INFO [GMFD:1] 2010-08-16 17:25:53,096 Gossiper.java (line 568)
 InetAddress /10.210.157.187 is now UP


 INFO [Thread-31] 2010-08-16 17:26:37,281 StreamCompletionHandler.java
 (line 64)
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-476-Data.db
 INFO [Timer-1] 2010-08-16 17:27:01,939 Gossiper.java (line 179)
 InetAddress /10.254.234.226 is now dead.
 INFO [GMFD:1] 2010-08-16 17:27:24,115 Gossiper.java (line 568)
 InetAddress /10.254.234.226 is now UP


 INFO [COMPACTION-POOL:1] 2010-08-16 18:05:08,764 CompactionManager.java
 (line 246) Compacting []
 INFO [COMPACTION-POOL:1] 2010-08-16 19:05:08,741 CompactionManager.java
 (line 246) Compacting []
 INFO [COMPACTION-POOL:1] 2010-08-16 20:05:08,718 CompactionManager.java
 (line 246) Compacting []
INFO [COMPACTION-POOL:1] 2010-08-16 21:05:08,694 CompactionManager.java
 (line 246) Compacting []
 INFO [COMPACTION-POOL:1] 2010-08-16 22:05:08,674 CompactionManager.java
 (line 246) Compacting []

******************************************************
At this point, I logged in.  The data distribution on this node was 122GB.  I
started performing a manual nodetool cleanup.
******************************************************





Mime
View raw message