incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ran Tavory <>
Subject server crash - how to invertigate
Date Wed, 14 Apr 2010 11:04:28 GMT
I'm running a 0.6.0 cluster with four nodes and one of them just crashed.

The logs all seem normal and I haven't seen anything special in the jmx
counters before the crash.

I have one client writing and reading using 10 threads and using 3 different
column families: KvAds, KvImpressions and KvUsers

the client had got a few UnavailableException, TimedOutException and
TTransportException but was able to complete the read/write operation by
failing over to another available host. I can't tell if the exceptions were
from the crashed host or from other hosts in the ring.

Any hints how to investigate this are greatly appreciated. So far I'm

Here's a snippet from the log just before it went down. It doesn't seem to
have anything special in it, everything is INFO level.

The only thing that seems a bit strange is that last message: Compacting [].
This message usually comes with things inside the [], such as Compacting
but this time it was just empty.
However, this is not the only place in the log were I see an empty
Compacting []. There are other places and they didn't end up in a crash, so
I don't know if it's related.

here's the log:
 INFO [ROW-MUTATION-STAGE:6] 2010-04-14 05:55:07,014
(line 357) KvImpressions has reached its threshold; switching in a fresh
Memtable at
 INFO [ROW-MUTATION-STAGE:6] 2010-04-14 05:55:07,015
(line 609) Enqueuing flush of Memtable(KvImpressions)@258729366
 INFO [FLUSH-WRITER-POOL:1] 2010-04-14 05:55:07,015 (line 148)
Writing Memtable(KvImpressions)@258729366
 INFO [FLUSH-WRITER-POOL:1] 2010-04-14 05:55:10,130 (line 162)
Completed flushing
 INFO [COMMIT-LOG-WRITER] 2010-04-14 05:55:10,154 (line 407)
Discarding obsolete commit
 INFO [SSTABLE-CLEANUP-TIMER] 2010-04-14 05:55:28,415 (line 104) Deleted
 INFO [SSTABLE-CLEANUP-TIMER] 2010-04-14 05:55:28,440 (line 104) Deleted
 INFO [SSTABLE-CLEANUP-TIMER] 2010-04-14 05:55:28,454 (line 104) Deleted
 INFO [SSTABLE-CLEANUP-TIMER] 2010-04-14 05:55:28,526 (line 104) Deleted
 INFO [SSTABLE-CLEANUP-TIMER] 2010-04-14 05:55:28,585 (line 104) Deleted
 INFO [SSTABLE-CLEANUP-TIMER] 2010-04-14 05:55:28,602 (line 104) Deleted
 INFO [SSTABLE-CLEANUP-TIMER] 2010-04-14 05:55:28,614 (line 104) Deleted
 INFO [SSTABLE-CLEANUP-TIMER] 2010-04-14 05:55:28,682 (line 104) Deleted
 INFO [COMMIT-LOG-WRITER] 2010-04-14 05:55:52,254
(line 50) Creating new commitlog segment
 INFO [ROW-MUTATION-STAGE:16] 2010-04-14 05:56:25,347
(line 357) KvImpressions has reached its threshold; switching in a fresh
Memtable at
 INFO [ROW-MUTATION-STAGE:16] 2010-04-14 05:56:25,348
(line 609) Enqueuing flush of Memtable(KvImpressions)@1955587316
 INFO [FLUSH-WRITER-POOL:1] 2010-04-14 05:56:25,348 (line 148)
Writing Memtable(KvImpressions)@1955587316
 INFO [FLUSH-WRITER-POOL:1] 2010-04-14 05:56:30,572 (line 162)
Completed flushing
 INFO [COMMIT-LOG-WRITER] 2010-04-14 05:57:26,790
(line 50) Creating new commitlog segment
 INFO [ROW-MUTATION-STAGE:7] 2010-04-14 05:57:59,513
(line 357) KvImpressions has reached its threshold; switching in a fresh
Memtable at
 INFO [ROW-MUTATION-STAGE:7] 2010-04-14 05:57:59,513
(line 609) Enqueuing flush of Memtable(KvImpressions)@1617250066
 INFO [FLUSH-WRITER-POOL:1] 2010-04-14 05:57:59,513 (line 148)
Writing Memtable(KvImpressions)@1617250066
 INFO [FLUSH-WRITER-POOL:1] 2010-04-14 05:58:04,604 (line 162)
Completed flushing
 INFO [COMPACTION-POOL:1] 2010-04-14 05:58:04,605
(line 246) Compacting
 INFO [COMMIT-LOG-WRITER] 2010-04-14 05:58:04,652 (line 407)
Discarding obsolete commit
 INFO [COMPACTION-POOL:1] 2010-04-14 05:58:25,975
(line 326) Compacted to
 458669533/442361674 bytes for 848169 keys.  Time: 21369ms.
 INFO [COMPACTION-POOL:1] 2010-04-14 05:59:14,913
(line 246) Compacting []


View raw message