incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nate McCall <zznat...@gmail.com>
Subject Re: too many warnings of Heap is full
Date Wed, 30 Jan 2013 20:51:41 GMT
What's the output of nodetool cfstats for those 2 column families on
cassNode2 and cassNode3? And what is the replication factor for this
cluster?

Per the previous reply, nodetool ring should show each of your nodes
with ~16.7% of the data if well balanced.

Also, the auto-detection for memory sizes in the startup script is a
little off w/r/t m1.xlarge because of the 'slightly less than 16gb' of
ram. It usually ends up allocating 4g/400m (max and young) whereas
8g/800m will give you some more breathing room.

On Wed, Jan 30, 2013 at 12:07 PM, Bryan Talbot <btalbot@aeriagames.com> wrote:
> My guess is that those one or two nodes with the gc pressure also have more
> rows in your big CF.  More rows could be due to imbalanced distribution if
> your'e not using a random partitioner or from those nodes not yet removing
> deleted rows which other nodes may have done.
>
> JVM heap space is used for a few things which scale with key count
> including:
> - bloom filter (for C* < 1.2)
> - index samples
>
> Other space is used but can be more easily controlled by tuning for
> - memtable
> - compaction
> - key cache
> - row cache
>
>
> So, if those nodes have more rows (check using "nodetool ring" or "nodetool
> cfstats") than the others you can try to:
> - reduce the number of rows by adding nodes, run manual / tune compactions
> to remove rows with expired tombstones, etc.
> - increase bloom filter fp chance
> - increase jvm heap size (don't go too big)
> - disable key or row cache
> - increase index sample interval
>
> Not all of those things are generally good especially to the extreme so
> don't go setting a 20 GB jvm heap without understanding the consequences for
> example.
>
> -Bryan
>
>
> On Wed, Jan 30, 2013 at 3:47 AM, Guillermo Barbero
> <guillermo.barbero@spotbros.com> wrote:
>>
>> Hi,
>>
>>   I'm viewing a weird behaviour in my cassandra cluster. Most of the
>> warning messages are due to Heap is % full. According to this link
>>
>> (http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassndra-1-0-6-GC-query-tt7323457.html)
>> there are two ways to "reduce pressure":
>> 1. Decrease the cache sizes
>> 2. Increase the index interval size
>>
>> Most of the flushes are in two column families (users and messages), I
>> guess that's because the most mutations are there.
>>
>> I still have not applied those changes to the production environment.
>> Do you recommend any other meassure? Should I set specific tunning for
>> these two CFs? Should I check another metric?
>>
>> Additionally, the distribution of warning messages is not uniform
>> along the cluster. Why could cassandra be doing this? What should I do
>> to find out how to fix this?
>>
>> cassandra runs on a 6 node cluster of m1.xlarge machines (Amazon EC2)
>> the java version is the following:
>> java version "1.6.0_37"
>> Java(TM) SE Runtime Environment (build 1.6.0_37-b06)
>> Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode)
>>
>> The cassandra system.log is resumed here (numer of messages, cassandra
>> node, class that reports the message, first word of the message)
>> 2013-01-26
>>       5 cassNode0: GCInspector.java Heap
>>       5 cassNode0: StorageService.java Flushing
>>     232 cassNode2: GCInspector.java Heap
>>     232 cassNode2: StorageService.java Flushing
>>     104 cassNode3: GCInspector.java Heap
>>     104 cassNode3: StorageService.java Flushing
>>       3 cassNode4: GCInspector.java Heap
>>       3 cassNode4: StorageService.java Flushing
>>       3 cassNode5: GCInspector.java Heap
>>       3 cassNode5: StorageService.java Flushing
>>
>> 2013-01-27
>>       2 cassNode0: GCInspector.java Heap
>>       2 cassNode0: StorageService.java Flushing
>>       3 cassNode1: GCInspector.java Heap
>>       3 cassNode1: StorageService.java Flushing
>>     189 cassNode2: GCInspector.java Heap
>>     189 cassNode2: StorageService.java Flushing
>>     104 cassNode3: GCInspector.java Heap
>>     104 cassNode3: StorageService.java Flushing
>>       1 cassNode4: GCInspector.java Heap
>>       1 cassNode4: StorageService.java Flushing
>>       1 cassNode5: GCInspector.java Heap
>>       1 cassNode5: StorageService.java Flushing
>>
>> 2013-01-28
>>       2 cassNode0: GCInspector.java Heap
>>       2 cassNode0: StorageService.java Flushing
>>       1 cassNode1: GCInspector.java Heap
>>       1 cassNode1: StorageService.java Flushing
>>       1 cassNode2: AutoSavingCache.java Reducing
>>     343 cassNode2: GCInspector.java Heap
>>     342 cassNode2: StorageService.java Flushing
>>     181 cassNode3: GCInspector.java Heap
>>     181 cassNode3: StorageService.java Flushing
>>       4 cassNode4: GCInspector.java Heap
>>       4 cassNode4: StorageService.java Flushing
>>       3 cassNode5: GCInspector.java Heap
>>       3 cassNode5: StorageService.java Flushing
>>
>> 2013-01-29
>>       2 cassNode0: GCInspector.java Heap
>>       2 cassNode0: StorageService.java Flushing
>>       3 cassNode1: GCInspector.java Heap
>>       3 cassNode1: StorageService.java Flushing
>>     156 cassNode2: GCInspector.java Heap
>>     156 cassNode2: StorageService.java Flushing
>>      71 cassNode3: GCInspector.java Heap
>>      71 cassNode3: StorageService.java Flushing
>>       2 cassNode4: GCInspector.java Heap
>>       2 cassNode4: StorageService.java Flushing
>>       2 cassNode5: GCInspector.java Heap
>>       1 cassNode5: Memtable.java setting
>>       2 cassNode5: StorageService.java Flushing
>>
>> --
>>
>> Guillermo Barbero - Backend Team
>>
>> Spotbros Technologies
>
>
>

Mime
View raw message