Since last night I am seeing CPU load spikes on our cassandra boxes(Occasionally load goes up to 20, its a Amazon EC2 c1.xlarge with 300 iops EBS). After digging around a little I believe its related to heap memory and flushing memtables.

From logs:
WARN 03:22:03,414 Heap is 0.7786981388910019 full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up to the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically

WARN 03:22:03,415 Flushing CFS(Keyspace='XXX', ColumnFamily='XXXX') to relieve memory pressure

I have three nodes and only 2 of them are hitting this high load, moreover cluster is under extremely light load, no one is using it since yesterday and I still see this load.

I also observed that `top -H` showed many threads in Sleep state and only a handful in R state. `nodetool cfstat` showed the following for the ColumnFamily in the above stated cassandra logs:

              Column Family: XXXX
SSTable count: 8
Space used (live): 1479005837
Space used (total): 1479005837
Number of Keys (estimate): 2923008
Memtable Columns Count: 35375
Memtable Data Size: 7088479
Memtable Switch Count: 2393
Read Count: 2339668632
Read Latency: 3.042 ms.
Write Count: 360448535
Write Latency: 0.079 ms.
Pending Tasks: 0
Bloom Filter False Positives: 143197
Bloom Filter False Ratio: 0.73004
Bloom Filter Space Used: 7142048
Compacted row minimum size: 73
Compacted row maximum size: **785939**
Compacted row mean size: 1957

`Compacted Row maximum size` for other ColumFamily is significantly less than this number. 

When starting this cluster we set 
> JVM_OPTS="$JVM_OPTS -Xss1000k"

We are using cassandra 1.1.0 and open-6-jdk

Can any one please help me understand whey with no load on the system I am still seeing such high load on my machines.