cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "MaHaiyang (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-3537) JVM heap is full and GC take no effect ,so many NonBlockingHashMap objects in heap
Date Fri, 02 Dec 2011 06:47:40 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161451#comment-13161451
] 

MaHaiyang commented on CASSANDRA-3537:
--------------------------------------

Heap memory can not be GC again in my lasted test , although I have set jamm correctly . Jmap
the memory ,find the same thing  ,too many Message objects and ralated objects  in the memory
! 

{code:xml} 
Object Histogram:

num 	  #instances	#bytes	Class description
--------------------------------------------------------------------------
1:		9101138	927302160	byte[]
2:		9998059	479906832	java.nio.HeapByteBuffer
3:		875398	221844128	long[]
4:		4302698	137686336	org.apache.cassandra.db.Column
5:		869859	137476712	java.lang.Object[]
6:		1794458	101193096	int[]
7:		1515578	60623120	java.math.BigInteger
8:		834628	46739168	org.cliffc.high_scale_lib.ConcurrentAutoTable$CAT
9:		1644330	39463920	java.util.concurrent.ConcurrentSkipListMap$Node
10:		1461787	35082888	org.apache.cassandra.db.DecoratedKey
11:		513576	24651648	java.util.Hashtable
12:		741037	24559960	char[]
13:		1512396	24198336	org.apache.cassandra.dht.BigIntegerToken
14:		741097	23715104	java.lang.String
15:		732184	17572416	org.apache.cassandra.io.sstable.IndexSummary$KeyPosition
16:		513594	15637344	java.util.Hashtable$Entry[]
17:		278209	15579704	org.cliffc.high_scale_lib.NonBlockingHashMap$CHM
18:		462641	14804512	java.net.Inet4Address
19:		578801	13891224	java.util.concurrent.ConcurrentSkipListMap$Index
20:		834627	13354032	org.cliffc.high_scale_lib.Counter
21:		556415	13353960	org.apache.cassandra.db.ReadResponse
22:		267483	12839184	org.apache.cassandra.service.ReadCallback
23:		513250	12318000	org.apache.cassandra.net.Message
24:		513248	12317952	org.apache.cassandra.net.Header
25:		501596	12038304	org.apache.cassandra.db.AbstractColumnContainer$DeletionInfo
26:		244644	11742912	org.apache.cassandra.db.ThreadSafeSortedColumns
27:		365024	11680768	java.util.concurrent.ConcurrentSkipListMap$HeadIndex
28:		464800	11155200	java.util.Collections$SynchronizedSet
29:		278209	11128360	org.cliffc.high_scale_lib.NonBlockingHashMap
30:		342741	10967712	org.apache.cassandra.utils.ExpiringMap$CacheableObject
31:		199478	9574944	org.apache.cassandra.db.SliceFromReadCommand
32:		278218	8902976	java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceFieldUpdaterImpl
33:		348179	8356296	org.apache.cassandra.db.ColumnFamily
34:		342741	8225784	org.apache.cassandra.net.CallbackInfo
35:		255241	8167712	org.apache.cassandra.db.ArrayBackedSortedColumns
36:		334724	8033376	java.util.ArrayList
37:		499963	7999408	java.util.concurrent.atomic.AtomicReference
38:		319779	7674696	org.apache.cassandra.db.Row
39:		187484	7499360	org.apache.cassandra.db.Memtable$6
40:		464794	7436704	java.util.Hashtable$KeySet
41:		187496	5999872	java.util.TreeMap$KeyIterator
42:		372748	5963968	java.util.concurrent.atomic.AtomicInteger
43:		236877	5685048	org.apache.cassandra.db.filter.QueryPath
44:		236733	5681592	org.apache.cassandra.service.RowDigestResolver
45:		300385	4806160	org.apache.cassandra.utils.SimpleCondition
46:		30726	4245920	* ConstMethodKlass
47:		30726	3696240	* MethodKlass
48:		151726	3641424	org.apache.cassandra.db.SuperColumn
49:		3134	3350536	* ConstantPoolKlass
50:		95622	3059904	java.util.concurrent.locks.ReentrantLock$NonfairSync
51:		45910	2497952	* SymbolKlass
52:		3134	2276040	* InstanceKlassKlass
53:		94544	2269056	java.util.concurrent.LinkedBlockingQueue$Node
54:		93130	2235120	java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
55:		133398	2134368	java.util.concurrent.ConcurrentSkipListMap$Values
56:		2813	2099368	* ConstantPoolCacheKlass
57:		37698	1809504	java.util.TreeMap
58:		37783	1511320	java.util.TreeMap$Entry
59:		93602	1497632	java.util.concurrent.locks.ReentrantLock
60:		30768	1476864	java.util.concurrent.LinkedBlockingQueue
61:		85156	1362496	java.util.concurrent.ConcurrentSkipListMap$EntrySet
62:		41451	1326432	org.apache.cassandra.service.RowRepairResolver
63:		41141	1316512	java.util.concurrent.ConcurrentHashMap$HashEntry
64:		32885	1315400	org.apache.cassandra.service.WriteResponseHandler
65:		3110	1277672	* MethodDataKlass
66:		39780	1272960	com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
67:		31556	1262240	org.apache.cassandra.net.AsyncResult
68:		30745	1229800	org.apache.cassandra.db.RangeSliceCommand
69:		37628	1204096	org.apache.cassandra.db.SliceByNamesReadCommand
70:		44528	1068672	com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$WeightedValue
71:		31974	1023168	java.util.concurrent.locks.AbstractQueuedSynchronizer$Node
72:		41435	994440	org.apache.cassandra.service.AsyncRepairCallback
73:		30821	986272	org.apache.cassandra.thrift.SliceRange
74:		30756	984192	java.util.RandomAccessSubList
75:		30745	983840	org.apache.cassandra.service.RangeSliceResponseResolver
76:		29115	931680	org.apache.cassandra.db.DeletedColumn
77:		21676	867040	org.apache.cassandra.db.ExpiringColumn
78:		35339	848136	java.lang.Long
79:		35204	844896	org.apache.cassandra.utils.Pair
80:		30924	742176	java.util.BitSet
81:		30821	739704	org.apache.cassandra.thrift.SlicePredicate
82:		30787	738888	org.apache.cassandra.dht.Bounds
83:		37673	602768	java.util.TreeSet
84:		37654	602464	java.util.TreeMap$KeySet
85:		31598	505568	java.util.concurrent.atomic.AtomicBoolean
86:		6575	420800	java.nio.DirectByteBufferR
87:		2004	416400	java.util.concurrent.ConcurrentHashMap$HashEntry[]
88:		3433	357032	java.lang.Class
89:		8976	287232	java.lang.ref.WeakReference
90:		4999	269008	short[]{code} 

                
> JVM heap is full and  GC take no effect ,so many NonBlockingHashMap objects in heap 
> ------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-3537
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3537
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.0.0
>         Environment: 当前堆大小: 
> 5,815,955 Kb
> 堆大小的最大值: 
> 6,045,696 Kb
> 分配的内存: 
> 6,045,696 Kb
> 暂挂结束操作: 
> 0 个对象
> 垃圾收集器: 
> Name = 'ParNew', Collections = 3,294, Total time spent = 2 minutes
> 垃圾收集器: 
> Name = 'ConcurrentMarkSweep', Collections = 5,909, Total time spent = 2 hours 17 minutes
>  
> 操作系统: 
> Linux 2.6.32.12-0.7-default
> 体系结构: 
> amd64
> 处理器的数目: 
> 16
> 分配的虚拟内存: 
> 42,748,416 Kb
> 物理内存总量: 
> 24,568,836 Kb
> 可用物理内存: 
>  7,136,380 Kb
> 交换空间总量: 
>  2,104,472 Kb
> 可用交换空间: 
>  1,970,800 Kb
>  
> VM 参数: 
> -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G -Xmn2400M -XX:+HeapDumpOnOutOfMemoryError
-Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=9000 -Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false -Dpasswd.properties=/opt/obs/cassandra/conf/passwd.properties
-Dpasswd.mode=MD5 -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true

> 类路径: 
> /opt/obs/cassandra/conf:/opt/obs/cassandra/build/classes/main:/opt/obs/cassandra/build/classes/thrift:/opt/obs/cassandra/lib/antlr-3.2.jar:/opt/obs/cassandra/lib/apache-cassandra-1.0.0.jar:/opt/obs/cassandra/lib/apache-cassandra-clientutil-1.0.0.jar:/opt/obs/cassandra/lib/apache-cassandra-thrift-1.0.0.jar:/opt/obs/cassandra/lib/avro-1.4.0-fixes.jar:/opt/obs/cassandra/lib/avro-1.4.0-sources-fixes.jar:/opt/obs/cassandra/lib/cassandra_simple_authentication.jar:/opt/obs/cassandra/lib/commons-cli-1.1.jar:/opt/obs/cassandra/lib/commons-codec-1.2.jar:/opt/obs/cassandra/lib/commons-lang-2.4.jar:/opt/obs/cassandra/lib/compress-lzf-0.8.4.jar:/opt/obs/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/obs/cassandra/lib/guava-r08.jar:/opt/obs/cassandra/lib/high-scale-lib-1.1.2.jar:/opt/obs/cassandra/lib/jackson-core-asl-1.4.0.jar:/opt/obs/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/opt/obs/cassandra/lib/jamm-0.2.5.jar:/opt/obs/cassandra/lib/jline-0.9.94.jar:/opt/obs/cassandra/lib/json-simple-1.1.jar:/opt/obs/cassandra/lib/libthrift-0.6.jar:/opt/obs/cassandra/lib/log4j-1.2.16.jar:/opt/obs/cassandra/lib/servlet-api-2.5-20081211.jar:/opt/obs/cassandra/lib/slf4j-api-1.6.1.jar:/opt/obs/cassandra/lib/slf4j-log4j12-1.6.1.jar:/opt/obs/cassandra/lib/snakeyaml-1.6.jar:/opt/obs/cassandra/lib/snappy-java-1.0.3.jar
>            Reporter: MaHaiyang
>
> I have 4 cassandra nodes ,and I put about 30G data to db for every nodes . It's just
4 days before I start the cluster ,but now every 4 nodes have the same problem ,JVM heap is
full  ,and  GC take no effect ,There must be some memory leak . Jmap the memory as follow:
> Object Histogram:
> num 	  #instances	#bytes	Class description
> --------------------------------------------------------------------------
> 1:		15793606	758093088	java.nio.HeapByteBuffer
> 2:		2153811	320138208	java.lang.Object[]
> 3:		6163192	197222144	org.apache.cassandra.db.Column
> 4:		2543836	175890256	int[]
> 5:		2168816	155397192	long[]
> 6:		2078123	116374888	org.cliffc.high_scale_lib.ConcurrentAutoTable$CAT
> 7:		1847111	73884440	java.math.BigInteger
> 8:		1234243	59243664	java.util.Hashtable
> 9:		1770829	58233000	char[]
> 10:		1770627	56660064	java.lang.String
> 11:		1665886	39981264	org.apache.cassandra.db.DecoratedKey
> 12:		692706	38791536	org.cliffc.high_scale_lib.NonBlockingHashMap$CHM
> 13:		1234274	37172088	java.util.Hashtable$Entry[]
> 14:		1133541	36273312	java.net.Inet4Address
> 15:		738528	35449344	org.apache.cassandra.service.ReadCallback
> 16:		2078118	33249888	org.cliffc.high_scale_lib.Counter
> 17:		1373886	32973264	org.apache.cassandra.db.ReadResponse
> 18:		1234023	29616552	org.apache.cassandra.net.Message
> 19:		1234019	29616456	org.apache.cassandra.net.Header
> 20:		1846185	29538960	org.apache.cassandra.dht.BigIntegerToken
> 21:		891378	28524096	org.apache.cassandra.utils.ExpiringMap$CacheableObject
> 22:		692706	27708240	org.cliffc.high_scale_lib.NonBlockingHashMap
> 23:		1148252	27558048	java.util.Collections$SynchronizedSet
> 24:		541977	26014896	org.apache.cassandra.db.SliceFromReadCommand
> 25:		998001	23952024	java.util.concurrent.ConcurrentSkipListMap$Node
> 26:		928792	22291008	java.util.ArrayList
> 27:		692715	22166880	java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceFieldUpdaterImpl
> 28:		891378	21393072	org.apache.cassandra.net.CallbackInfo
> 29:		1148247	18371952	java.util.Hashtable$KeySet
> 30:		731859	17564616	org.apache.cassandra.db.Row
> 31:		529991	16959712	org.apache.cassandra.db.ArrayBackedSortedColumns
> 32:		691425	16594200	org.apache.cassandra.db.AbstractColumnContainer$DeletionInfo
> 33:		648580	15565920	org.apache.cassandra.db.filter.QueryPath
> 34:		648338	15560112	org.apache.cassandra.service.RowDigestResolver
> 35:		971376	15542016	java.util.concurrent.atomic.AtomicInteger
> 36:		837418	13398688	org.apache.cassandra.utils.SimpleCondition
> 37:		535614	12854736	org.apache.cassandra.db.ColumnFamily
> 38:		725634	11610144	java.util.concurrent.atomic.AtomicReference
> 39:		195117	9365616	org.apache.cassandra.db.ThreadSafeSortedColumns
> 40:		281921	9021472	java.util.concurrent.ConcurrentSkipListMap$HeadIndex
> 41:		277679	8885728	java.util.concurrent.locks.ReentrantLock$NonfairSync
> 42:		314424	7546176	java.util.concurrent.ConcurrentSkipListMap$Index
> 43:		275186	6604464	java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
> 44:		270280	6486720	java.util.concurrent.LinkedBlockingQueue$Node
> 45:		219553	5269272	org.apache.cassandra.io.sstable.IndexSummary$KeyPosition
> 46:		106436	5108928	java.util.TreeMap
> 47:		122185	4887400	org.apache.cassandra.db.ExpiringColumn
> 48:		189968	4559232	org.apache.cassandra.db.SuperColumn
> 49:		275659	4410544	java.util.concurrent.locks.ReentrantLock
> 50:		90213	4330224	java.util.concurrent.LinkedBlockingQueue
> 51:		107026	4281040	java.util.TreeMap$Entry
> 52:		30501	4222056	* ConstMethodKlass

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message