We have 20 nodes in this cluster.
Yes, however are you recommending that I decommission the node?
I noted the compaction because it is common for the last line in the log file. For reference:
INFO [FlushWriter:12] 2011-10-12 18:10:09,823 Memtable.java (line 157) Writing Memtable-HintsColumnFamily@
1313157506(0 bytes, 1 operations)
INFO [CompactionExecutor:1] 2011-10-12 18:10:09,823 CompactionManager.java (line 395) Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily-f-7958-Data.db'),SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily-f-7959-Data.db'),SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily-f-7960-Data.db'),SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily-f-7961-Data.db')]
INFO [FlushWriter:12] 2011-10-12 18:10:09,862 Memtable.java (line 172) Completed flushing /var/lib/cassandra/data/system/HintsColumnFamily-f-7962-Data.db (61 bytes)
Load, cpu and memory are nominal. The box is not stressed. iostat reports low usage.
On Wed, Oct 12, 2011 at 12:24 PM, Mohit Anchlia <email@example.com>
You mentioned this happens only on one node? How many nodes do you
have? Is it possible to turn off this node completely and run
compactions on other nodes and see if this happens there too?
Also, you mentioned this happens after compaction. Did you mean during
compaction or right after it? What load, cpu, memory etc do you see
during those times?