cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ipremyadav <ipremya...@gmail.com>
Subject Re: cassandra full gc too often
Date Thu, 31 Dec 2015 16:16:44 GMT
Simplest option is to use java 8 with G1 gc. 

> On 31 Dec 2015, at 10:23 a.m., Shuo Chen <chenatu2006@gmail.com> wrote:
> 
> I have a cassandra 2.0.6 cluster with four nodes as backup database. The only operation
is posting data into db. Recently, the full gc of the nodes increases apparently and blocks
cluster operation.
> 
> The load of each node is 10G. The heap is 8G each with default jvm memory settings. The
cpu is 24 cores. The settings of cassandra.yaml are almost default.
> 
> For debug, all the clients are disconnected, however the gc is still high.
> 
> The GC is too often that it appears in every minute and blocks for nealy 30s.
> 
> How to debug this situation
> 
> The following is the snippets of gc log:
> 
> before full gc histogram
> 
> 
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> 1620.992: [GC 7377342K(8178944K), 0.1380260 secs]
> Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> ----------------------------------------------
>    1:      61030245     1952967840  java.util.concurrent.FutureTask
>    2:      61031398     1464753552  java.util.concurrent.Executors$RunnableAdapter
>    3:      61030312     1464727488  java.util.concurrent.LinkedBlockingQueue$Node
>    4:      61030244     1464725856  org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask
>    5:        214181       13386992  [B
>    6:        202890        9738720  java.nio.HeapByteBuffer
>    7:         37622        5994808  [C
>    8:         42163        5722096  <constMethodKlass>
>    9:         42163        5407152  <methodKlass>
>   10:          4170        4648240  <constantPoolKlass>
>   11:        100060        4002400  org.apache.cassandra.io.sstable.IndexHelper$IndexInfo
>   12:          4170        2860816  <instanceKlassKlass>
>   13:          3664        2702720  <constantPoolCacheKlass>
>   14:          4817        2701576  [J
>   15:          2575        1056600  <methodDataKlass>
>   16:         37431         898344  java.lang.String
>   17:          2585         627352  [Ljava.lang.Object;
>   18:         17728         567296  java.util.concurrent.ConcurrentHashMap$HashEntry
>   19:         22111         530664  javax.management.ObjectName$Property
>   20:          7595         463824  [S
>   21:          4534         433560  java.lang.Class
>   22:         11581         370592  java.util.HashMap$Entry
> 
> ...
> Total     289942930     8399196504
> 1658.022: [Full GC 8178943K->6241829K(8178944K), 27.8562520 secs]
> CMS: Large block 0x00000007f7da9588
> 
> After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 6335823
> Max   Chunk Size: 6335823
> Number of Blocks: 1
> Av.  Block  Size: 6335823
> Tree      Height: 1
> After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> 
> 
> It seems related to objects of Futuretask.
> 
> -- 
> 陈硕 Shuo Chen
> chenatu2006@gmail.com
> chenshuo@whaty.com

Mime
View raw message