looks like reads are backing up, which in turn is making deserialize back up
On Sun, May 23, 2010 at 4:25 AM, Ran Tavory <rantav@gmail.com> wrote:
> Here's tpstats on a server with traffic that I think will get OOM shortly.
> We have 4k pending reads and 123k pending at MESSAGE-DESERIALIZER-POOL
> Is there something I can do to prevent that? (other than adding RAM...)
> Pool Name Active Pending Completed
> FILEUTILS-DELETE-POOL 0 0 55
> STREAM-STAGE 0 0 6
> RESPONSE-STAGE 0 0 0
> ROW-READ-STAGE 8 4088 7537229
> LB-OPERATIONS 0 0 0
> MESSAGE-DESERIALIZER-POOL 1 123799 22198459
> GMFD 0 0 471827
> LB-TARGET 0 0 0
> CONSISTENCY-MANAGER 0 0 0
> ROW-MUTATION-STAGE 0 0 14142351
> MESSAGE-STREAMING-POOL 0 0 16
> LOAD-BALANCER-STAGE 0 0 0
> FLUSH-SORTER-POOL 0 0 0
> MEMTABLE-POST-FLUSHER 0 0 128
> FLUSH-WRITER-POOL 0 0 128
> AE-SERVICE-STAGE 1 1 8
> HINTED-HANDOFF-POOL 0 0 10
>
> On Sat, May 22, 2010 at 11:05 PM, Ran Tavory <rantav@gmail.com> wrote:
>>
>> The message deserializer has 10m pending tasks before the oom. What do you
>> think makes the message deserializer blow up? I'd suspect that when it goes
>> up to 10m pending tasks, don't know how much mem a task actually takes up,
>> but they may consume a lot of memory. Is there a setting I need to tweak?
>> (or am I barking at the wrong tree?).
>> I'll add the counters
>> from http://github.com/jbellis/cassandra-munin-plugins but I already have
>> most of them monitored, so I attached the graphs of the ones that seemed the
>> most suspicious in the previous email.
>> The system keyspace and HH CF don't look too bad, I think, here they are:
>> Keyspace: system
>> Read Count: 154
>> Read Latency: 0.875012987012987 ms.
>> Write Count: 9
>> Write Latency: 0.20055555555555554 ms.
>> Pending Tasks: 0
>> Column Family: LocationInfo
>> SSTable count: 1
>> Space used (live): 2714
>> Space used (total): 2714
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 3
>> Read Count: 2
>> Read Latency: NaN ms.
>> Write Count: 9
>> Write Latency: 0.011 ms.
>> Pending Tasks: 0
>> Key cache capacity: 1
>> Key cache size: 1
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 203
>> Compacted row maximum size: 397
>> Compacted row mean size: 300
>> Column Family: HintsColumnFamily
>> SSTable count: 1
>> Space used (live): 1457
>> Space used (total): 4371
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 0
>> Read Count: 152
>> Read Latency: 0.369 ms.
>> Write Count: 0
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 1
>> Key cache size: 1
>> Key cache hit rate: 0.07142857142857142
>> Row cache: disabled
>> Compacted row minimum size: 829
>> Compacted row maximum size: 829
>> Compacted row mean size: 829
>>
>>
>>
>>
>> On Sat, May 22, 2010 at 4:14 AM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>>
>>> Can you monitor cassandra-level metrics like the ones in
>>> http://github.com/jbellis/cassandra-munin-plugins ?
>>>
>>> the usual culprit is usually compaction but your compacted row size is
>>> small. nothing else really comes to mind.
>>>
>>> (you should check system keyspace too tho, HH rows can get large)
>>>
>>> On Fri, May 21, 2010 at 2:36 PM, Ran Tavory <rantav@gmail.com> wrote:
>>> > I see some OOM on one of the hosts in the cluster and I wonder if
>>> > there's a
>>> > formula that'll help me calculate what's the required memory setting
>>> > given
>>> > the parameters x,y,z...
>>> > In short, I need advice on:
>>> > 1. How to set up proper heap space and which parameters should I look
>>> > at
>>> > when doing so.
>>> > 2. Help setting up an alert policy and define some counter measures or
>>> > sos
>>> > steps an admin can take to prevent further degradation of service when
>>> > alerts fire.
>>> > The OOM is at the row mutation stage and it happens after extensive GC
>>> > activity. (log tail below).
>>> > The server has 16G physical ram and java heap space 4G. No other
>>> > significant
>>> > processes run on the same server. I actually upped the java heap space
>>> > to 8G
>>> > but it OOMed again...
>>> > Most of my settings are the defaults with a few keyspaces and a few CFs
>>> > in
>>> > each KS. Here's the output of cfstats for the largest and most heavily
>>> > used
>>> > CF. (currently reads/writes are stopped but data is there).
>>> > Keyspace: outbrain_kvdb
>>> > Read Count: 3392
>>> > Read Latency: 160.33135908018866 ms.
>>> > Write Count: 2005839
>>> > Write Latency: 0.029233923061621595 ms.
>>> > Pending Tasks: 0
>>> > Column Family: KvImpressions
>>> > SSTable count: 8
>>> > Space used (live): 21923629878
>>> > Space used (total): 21923629878
>>> > Memtable Columns Count: 69440
>>> > Memtable Data Size: 9719364
>>> > Memtable Switch Count: 26
>>> > Read Count: 3392
>>> > Read Latency: NaN ms.
>>> > Write Count: 1998821
>>> > Write Latency: 0.018 ms.
>>> > Pending Tasks: 0
>>> > Key cache capacity: 200000
>>> > Key cache size: 11661
>>> > Key cache hit rate: NaN
>>> > Row cache: disabled
>>> > Compacted row minimum size: 302
>>> > Compacted row maximum size: 22387
>>> > Compacted row mean size: 641
>>> > I'm also attaching a few graphs of "the incidenst" I hope they help.
>>> > From
>>> > the graphs it looks like:
>>> > 1. message deserializer pool is behind so maybe taking too much mem. If
>>> > graphs are correct, it gets as high as 10m pending before crash.
>>> > 2. row-read-stage has a high number of pending (4k) so first of all -
>>> > this
>>> > isn't good for performance whether it caused the oom or not, and
>>> > second,
>>> > this may also have taken up heap space and caused the crash.
>>> > Thanks!
>>> > INFO [GC inspection] 2010-05-21 00:53:25,885 GCInspector.java (line
>>> > 110) GC
>>> > for ConcurrentMarkSweep: 10819 ms, 939992 reclaimed leaving 4312064504
>>> > used;
>>> > max is 4431216640
>>> > INFO [GC inspection] 2010-05-21 00:53:44,605 GCInspector.java (line
>>> > 110) GC
>>> > for ConcurrentMarkSweep: 9672 ms, 673400 reclaimed leaving 4312337208
>>> > used;
>>> > max is 4431216640
>>> > INFO [GC inspection] 2010-05-21 00:54:23,110 GCInspector.java (line
>>> > 110) GC
>>> > for ConcurrentMarkSweep: 9150 ms, 402072 reclaimed leaving 4312609776
>>> > used;
>>> > max is 4431216640
>>> > ERROR [ROW-MUTATION-STAGE:19] 2010-05-21 01:55:37,951
>>> > CassandraDaemon.java
>>> > (line 88) Fatal exception in thread
>>> > Thread[ROW-MUTATION-STAGE:19,5,main]
>>> > java.lang.OutOfMemoryError: Java heap space
>>> > ERROR [Thread-10] 2010-05-21 01:55:37,951 CassandraDaemon.java (line
>>> > 88)
>>> > Fatal exception in thread Thread[Thread-10,5,main]
>>> > java.lang.OutOfMemoryError: Java heap space
>>> > ERROR [CACHETABLE-TIMER-2] 2010-05-21 01:55:37,951 CassandraDaemon.java
>>> > (line 88) Fatal exception in thread Thread[CACHETABLE-TIMER-2,5,main]
>>> > java.lang.OutOfMemoryError: Java heap space
>>> >
>>>
>>>
>>>
>>> --
>>> Jonathan Ellis
>>> Project Chair, Apache Cassandra
>>> co-founder of Riptano, the source for professional Cassandra support
>>> http://riptano.com
>>
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
|