Here's tpstats on a server with traffic that I think will get OOM shortly. 
We have 4k pending reads and 123k pending at MESSAGE-DESERIALIZER-POOL

Is there something I can do to prevent that? (other than adding RAM...)

Pool Name                    Active   Pending      Completed
FILEUTILS-DELETE-POOL             0         0             55
STREAM-STAGE                      0         0              6
RESPONSE-STAGE                    0         0              0
ROW-READ-STAGE                    8      4088        7537229
LB-OPERATIONS                     0         0              0
MESSAGE-DESERIALIZER-POOL         1    123799       22198459
GMFD                              0         0         471827
LB-TARGET                         0         0              0
CONSISTENCY-MANAGER               0         0              0
ROW-MUTATION-STAGE                0         0       14142351
MESSAGE-STREAMING-POOL            0         0             16
LOAD-BALANCER-STAGE               0         0              0
FLUSH-SORTER-POOL                 0         0              0
MEMTABLE-POST-FLUSHER             0         0            128
FLUSH-WRITER-POOL                 0         0            128
AE-SERVICE-STAGE                  1         1              8
HINTED-HANDOFF-POOL               0         0             10


On Sat, May 22, 2010 at 11:05 PM, Ran Tavory <rantav@gmail.com> wrote:
The message deserializer has 10m pending tasks before the oom. What do you think makes the message deserializer blow up? I'd suspect that when it goes up to 10m pending tasks, don't know how much mem a task actually takes up, but they may consume a lot of memory. Is there a setting I need to tweak? (or am I barking at the wrong tree?).

I'll add the counters from http://github.com/jbellis/cassandra-munin-plugins but I already have most of them monitored, so I attached the graphs of the ones that seemed the most suspicious in the previous email.

The system keyspace and HH CF don't look too bad, I think, here they are:

Keyspace: system
        Read Count: 154
        Read Latency: 0.875012987012987 ms.
        Write Count: 9
        Write Latency: 0.20055555555555554 ms.
        Pending Tasks: 0
                Column Family: LocationInfo
                SSTable count: 1
                Space used (live): 2714
                Space used (total): 2714
                Memtable Columns Count: 0
                Memtable Data Size: 0
                Memtable Switch Count: 3
                Read Count: 2
                Read Latency: NaN ms.
                Write Count: 9
                Write Latency: 0.011 ms.
                Pending Tasks: 0
                Key cache capacity: 1
                Key cache size: 1
                Key cache hit rate: NaN
                Row cache: disabled
                Compacted row minimum size: 203
                Compacted row maximum size: 397
                Compacted row mean size: 300

                Column Family: HintsColumnFamily
                SSTable count: 1
                Space used (live): 1457
                Space used (total): 4371
                Memtable Columns Count: 0
                Memtable Data Size: 0
                Memtable Switch Count: 0
                Read Count: 152
                Read Latency: 0.369 ms.
                Write Count: 0
                Write Latency: NaN ms.
                Pending Tasks: 0
                Key cache capacity: 1
                Key cache size: 1
                Key cache hit rate: 0.07142857142857142
                Row cache: disabled
                Compacted row minimum size: 829
                Compacted row maximum size: 829
                Compacted row mean size: 829





On Sat, May 22, 2010 at 4:14 AM, Jonathan Ellis <jbellis@gmail.com> wrote:
Can you monitor cassandra-level metrics like the ones in
http://github.com/jbellis/cassandra-munin-plugins ?

the usual culprit is usually compaction but your compacted row size is
small.  nothing else really comes to mind.

(you should check system keyspace too tho, HH rows can get large)

On Fri, May 21, 2010 at 2:36 PM, Ran Tavory <rantav@gmail.com> wrote:
> I see some OOM on one of the hosts in the cluster and I wonder if there's a
> formula that'll help me calculate what's the required memory setting given
> the parameters x,y,z...
> In short, I need advice on:
> 1. How to set up proper heap space and which parameters should I look at
> when doing so.
> 2. Help setting up an alert policy and define some counter measures or sos
> steps an admin can take to prevent further degradation of service when
> alerts fire.
> The OOM is at the row mutation stage and it happens after extensive GC
> activity. (log tail below).
> The server has 16G physical ram and java heap space 4G. No other significant
> processes run on the same server. I actually upped the java heap space to 8G
> but it OOMed again...
> Most of my settings are the defaults with a few keyspaces and a few CFs in
> each KS. Here's the output of cfstats for the largest and most heavily used
> CF. (currently reads/writes are stopped but data is there).
> Keyspace: outbrain_kvdb
>         Read Count: 3392
>         Read Latency: 160.33135908018866 ms.
>         Write Count: 2005839
>         Write Latency: 0.029233923061621595 ms.
>         Pending Tasks: 0
>                 Column Family: KvImpressions
>                 SSTable count: 8
>                 Space used (live): 21923629878
>                 Space used (total): 21923629878
>                 Memtable Columns Count: 69440
>                 Memtable Data Size: 9719364
>                 Memtable Switch Count: 26
>                 Read Count: 3392
>                 Read Latency: NaN ms.
>                 Write Count: 1998821
>                 Write Latency: 0.018 ms.
>                 Pending Tasks: 0
>                 Key cache capacity: 200000
>                 Key cache size: 11661
>                 Key cache hit rate: NaN
>                 Row cache: disabled
>                 Compacted row minimum size: 302
>                 Compacted row maximum size: 22387
>                 Compacted row mean size: 641
> I'm also attaching a few graphs of "the incidenst" I hope they help. From
> the graphs it looks like:
> 1. message deserializer pool is behind so maybe taking too much mem. If
> graphs are correct, it gets as high as 10m pending before crash.
> 2. row-read-stage has a high number of pending (4k) so first of all - this
> isn't good for performance whether it caused the oom or not, and second,
> this may also have taken up heap space and caused the crash.
> Thanks!
>  INFO [GC inspection] 2010-05-21 00:53:25,885 GCInspector.java (line 110) GC
> for ConcurrentMarkSweep: 10819 ms, 939992 reclaimed leaving 4312064504 used;
> max is 4431216640
>  INFO [GC inspection] 2010-05-21 00:53:44,605 GCInspector.java (line 110) GC
> for ConcurrentMarkSweep: 9672 ms, 673400 reclaimed leaving 4312337208 used;
> max is 4431216640
>  INFO [GC inspection] 2010-05-21 00:54:23,110 GCInspector.java (line 110) GC
> for ConcurrentMarkSweep: 9150 ms, 402072 reclaimed leaving 4312609776 used;
> max is 4431216640
> ERROR [ROW-MUTATION-STAGE:19] 2010-05-21 01:55:37,951 CassandraDaemon.java
> (line 88) Fatal exception in thread Thread[ROW-MUTATION-STAGE:19,5,main]
> java.lang.OutOfMemoryError: Java heap space
> ERROR [Thread-10] 2010-05-21 01:55:37,951 CassandraDaemon.java (line 88)
> Fatal exception in thread Thread[Thread-10,5,main]
> java.lang.OutOfMemoryError: Java heap space
> ERROR [CACHETABLE-TIMER-2] 2010-05-21 01:55:37,951 CassandraDaemon.java
> (line 88) Fatal exception in thread Thread[CACHETABLE-TIMER-2,5,main]
> java.lang.OutOfMemoryError: Java heap space
>



--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com