cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shuo Chen <chenatu2...@gmail.com>
Subject Re: cassandra full gc too often
Date Mon, 04 Jan 2016 01:54:34 GMT
The attachment is the current system.log and config of the cassandra
cluster. Over 8000 full gc happens in 3 days with 121 young gc.

There are client operation in these days. Besides most columnfamily in the
cluster are supercolumnfamily created by cassandra-cli. Most rows have
average 30 sub-rows and each sub-row has 20 columns.

You said that there are many compaction tasks. So how to check the source
of this compactions? Thank you!

On Fri, Jan 1, 2016 at 10:09 AM, Graham Sanderson <graham@vast.com> wrote:

> If you are lucky that might mask the real issue, but I doubt it… that is
> an insane number of compaction tasks and indicative of another problem. I
> would check release notes of 2.0.6+, if I recall that was not a stable
> version and may have had leaks.
>
> Aside from that, just FYI, if you use native_objects for memtables you can
> run CMS fine even up to largish (20-30G) heap sizes without long GC pauses
> (at least over months) . That said look for promotion failures of 131074
> DWORDS. Not sure what your GC logging options are, but what you posted
> doesn’t say why a full GC happened which would be totally tunable, but as I
> say I don’t think GC is actually your problem
>
>
> On Dec 31, 2015, at 10:16 AM, Ipremyadav <ipremyadav@gmail.com> wrote:
>
> Simplest option is to use java 8 with G1 gc.
>
> On 31 Dec 2015, at 10:23 a.m., Shuo Chen <chenatu2006@gmail.com> wrote:
>
> I have a cassandra 2.0.6 cluster with four nodes as backup database. The
> only operation is posting data into db. Recently, the full gc of the nodes
> increases apparently and blocks cluster operation.
>
> The load of each node is 10G. The heap is 8G each with default jvm memory
> settings. The cpu is 24 cores. The settings of cassandra.yaml are almost
> default.
>
> For debug, all the clients are disconnected, however the gc is still high.
>
> The GC is too often that it appears in every minute and blocks for nealy
> 30s.
>
> How to debug this situation
>
> The following is the snippets of gc log:
>
> before full gc histogram
>
>
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> 1620.992: [GC 7377342K(8178944K), 0.1380260 secs]
> Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> ----------------------------------------------
>    1:      61030245     1952967840  java.util.concurrent.FutureTask
>    2:      61031398     1464753552
>  java.util.concurrent.Executors$RunnableAdapter
>    3:      61030312     1464727488
>  java.util.concurrent.LinkedBlockingQueue$Node
>    4:      61030244     1464725856
>  org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask
>    5:        214181       13386992  [B
>    6:        202890        9738720  java.nio.HeapByteBuffer
>    7:         37622        5994808  [C
>    8:         42163        5722096  <constMethodKlass>
>    9:         42163        5407152  <methodKlass>
>   10:          4170        4648240  <constantPoolKlass>
>   11:        100060        4002400
>  org.apache.cassandra.io.sstable.IndexHelper$IndexInfo
>   12:          4170        2860816  <instanceKlassKlass>
>   13:          3664        2702720  <constantPoolCacheKlass>
>   14:          4817        2701576  [J
>   15:          2575        1056600  <methodDataKlass>
>   16:         37431         898344  java.lang.String
>   17:          2585         627352  [Ljava.lang.Object;
>   18:         17728         567296
>  java.util.concurrent.ConcurrentHashMap$HashEntry
>   19:         22111         530664  javax.management.ObjectName$Property
>   20:          7595         463824  [S
>   21:          4534         433560  java.lang.Class
>   22:         11581         370592  java.util.HashMap$Entry
>
> ...
> Total     289942930     8399196504
> 1658.022: [Full GC 8178943K->6241829K(8178944K), 27.8562520 secs]
> CMS: Large block 0x00000007f7da9588
>
> After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 6335823
> Max   Chunk Size: 6335823
> Number of Blocks: 1
> Av.  Block  Size: 6335823
> Tree      Height: 1
> After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
>
>
> It seems related to objects of Futuretask.
>
> --
> *陈硕* *Shuo Chen*
> chenatu2006@gmail.com
> chenshuo@whaty.com
>
>
>


-- 
*陈硕* *Shuo Chen*
chenatu2006@gmail.com
chenshuo@whaty.com

Mime
View raw message