My guess is that those one or two nodes with the gc pressure also have more rows in your big CF.  More rows could be due to imbalanced distribution if your'e not using a random partitioner or from those nodes not yet removing deleted rows which other nodes may have done.

JVM heap space is used for a few things which scale with key count including:
- bloom filter (for C* < 1.2)
- index samples

Other space is used but can be more easily controlled by tuning for
- memtable
- compaction
- key cache
- row cache


So, if those nodes have more rows (check using "nodetool ring" or "nodetool cfstats") than the others you can try to:
- reduce the number of rows by adding nodes, run manual / tune compactions to remove rows with expired tombstones, etc.
- increase bloom filter fp chance
- increase jvm heap size (don't go too big)
- disable key or row cache
- increase index sample interval

Not all of those things are generally good especially to the extreme so don't go setting a 20 GB jvm heap without understanding the consequences for example.

-Bryan


On Wed, Jan 30, 2013 at 3:47 AM, Guillermo Barbero <guillermo.barbero@spotbros.com> wrote:
Hi,

  I'm viewing a weird behaviour in my cassandra cluster. Most of the
warning messages are due to Heap is % full. According to this link
(http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassndra-1-0-6-GC-query-tt7323457.html)
there are two ways to "reduce pressure":
1. Decrease the cache sizes
2. Increase the index interval size

Most of the flushes are in two column families (users and messages), I
guess that's because the most mutations are there.

I still have not applied those changes to the production environment.
Do you recommend any other meassure? Should I set specific tunning for
these two CFs? Should I check another metric?

Additionally, the distribution of warning messages is not uniform
along the cluster. Why could cassandra be doing this? What should I do
to find out how to fix this?

cassandra runs on a 6 node cluster of m1.xlarge machines (Amazon EC2)
the java version is the following:
java version "1.6.0_37"
Java(TM) SE Runtime Environment (build 1.6.0_37-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode)

The cassandra system.log is resumed here (numer of messages, cassandra
node, class that reports the message, first word of the message)
2013-01-26
      5 cassNode0: GCInspector.java Heap
      5 cassNode0: StorageService.java Flushing
    232 cassNode2: GCInspector.java Heap
    232 cassNode2: StorageService.java Flushing
    104 cassNode3: GCInspector.java Heap
    104 cassNode3: StorageService.java Flushing
      3 cassNode4: GCInspector.java Heap
      3 cassNode4: StorageService.java Flushing
      3 cassNode5: GCInspector.java Heap
      3 cassNode5: StorageService.java Flushing

2013-01-27
      2 cassNode0: GCInspector.java Heap
      2 cassNode0: StorageService.java Flushing
      3 cassNode1: GCInspector.java Heap
      3 cassNode1: StorageService.java Flushing
    189 cassNode2: GCInspector.java Heap
    189 cassNode2: StorageService.java Flushing
    104 cassNode3: GCInspector.java Heap
    104 cassNode3: StorageService.java Flushing
      1 cassNode4: GCInspector.java Heap
      1 cassNode4: StorageService.java Flushing
      1 cassNode5: GCInspector.java Heap
      1 cassNode5: StorageService.java Flushing

2013-01-28
      2 cassNode0: GCInspector.java Heap
      2 cassNode0: StorageService.java Flushing
      1 cassNode1: GCInspector.java Heap
      1 cassNode1: StorageService.java Flushing
      1 cassNode2: AutoSavingCache.java Reducing
    343 cassNode2: GCInspector.java Heap
    342 cassNode2: StorageService.java Flushing
    181 cassNode3: GCInspector.java Heap
    181 cassNode3: StorageService.java Flushing
      4 cassNode4: GCInspector.java Heap
      4 cassNode4: StorageService.java Flushing
      3 cassNode5: GCInspector.java Heap
      3 cassNode5: StorageService.java Flushing

2013-01-29
      2 cassNode0: GCInspector.java Heap
      2 cassNode0: StorageService.java Flushing
      3 cassNode1: GCInspector.java Heap
      3 cassNode1: StorageService.java Flushing
    156 cassNode2: GCInspector.java Heap
    156 cassNode2: StorageService.java Flushing
     71 cassNode3: GCInspector.java Heap
     71 cassNode3: StorageService.java Flushing
      2 cassNode4: GCInspector.java Heap
      2 cassNode4: StorageService.java Flushing
      2 cassNode5: GCInspector.java Heap
      1 cassNode5: Memtable.java setting
      2 cassNode5: StorageService.java Flushing

--

Guillermo Barbero - Backend Team

Spotbros Technologies