cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "MaHaiyang (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-3537) JVM heap is full and GC take no effect ,so many NonBlockingHashMap objects in heap
Date Fri, 02 Dec 2011 02:04:43 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161363#comment-13161363
] 

MaHaiyang commented on CASSANDRA-3537:
--------------------------------------

After  I set jamm correctly , I do a new test , and haven't find the memory problem in 20
hours  .But I could't confirm it's  "jamm not specified as java agent"  result the heap problem
. I analysised the dump memory info of the node which had the heap problem , they nearly have
the same thing :  too many Message objects and ralated objects like below,although I have
stoped reading and writing data for a few minutes and  empty all memtables  .
#instances	#bytes	Class description
---------------------------------------------
900654	21615696	org.apache.cassandra.net.Message
900653	21615672	org.apache.cassandra.net.Header
341351	16384848	org.apache.cassandra.service.ReadCallback
338502	13540080	org.apache.cassandra.db.RangeSliceCommand
338500	10832000	org.apache.cassandra.thrift.SliceRange
338502	8124048	org.apache.cassandra.thrift.SlicePredicate
343328	5493248	org.apache.cassandra.utils.SimpleCondition


Is there someone can tell me ,whether there are some relationship between "jamm not specified
as java agent" and so many 
objects like above  ?
 
                
> JVM heap is full and  GC take no effect ,so many NonBlockingHashMap objects in heap 
> ------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-3537
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3537
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.0.0
>         Environment: 当前堆大小: 
> 5,815,955 Kb
> 堆大小的最大值: 
> 6,045,696 Kb
> 分配的内存: 
> 6,045,696 Kb
> 暂挂结束操作: 
> 0 个对象
> 垃圾收集器: 
> Name = 'ParNew', Collections = 3,294, Total time spent = 2 minutes
> 垃圾收集器: 
> Name = 'ConcurrentMarkSweep', Collections = 5,909, Total time spent = 2 hours 17 minutes
>  
> 操作系统: 
> Linux 2.6.32.12-0.7-default
> 体系结构: 
> amd64
> 处理器的数目: 
> 16
> 分配的虚拟内存: 
> 42,748,416 Kb
> 物理内存总量: 
> 24,568,836 Kb
> 可用物理内存: 
>  7,136,380 Kb
> 交换空间总量: 
>  2,104,472 Kb
> 可用交换空间: 
>  1,970,800 Kb
>  
> VM 参数: 
> -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G -Xmn2400M -XX:+HeapDumpOnOutOfMemoryError
-Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=9000 -Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false -Dpasswd.properties=/opt/obs/cassandra/conf/passwd.properties
-Dpasswd.mode=MD5 -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true

> 类路径: 
> /opt/obs/cassandra/conf:/opt/obs/cassandra/build/classes/main:/opt/obs/cassandra/build/classes/thrift:/opt/obs/cassandra/lib/antlr-3.2.jar:/opt/obs/cassandra/lib/apache-cassandra-1.0.0.jar:/opt/obs/cassandra/lib/apache-cassandra-clientutil-1.0.0.jar:/opt/obs/cassandra/lib/apache-cassandra-thrift-1.0.0.jar:/opt/obs/cassandra/lib/avro-1.4.0-fixes.jar:/opt/obs/cassandra/lib/avro-1.4.0-sources-fixes.jar:/opt/obs/cassandra/lib/cassandra_simple_authentication.jar:/opt/obs/cassandra/lib/commons-cli-1.1.jar:/opt/obs/cassandra/lib/commons-codec-1.2.jar:/opt/obs/cassandra/lib/commons-lang-2.4.jar:/opt/obs/cassandra/lib/compress-lzf-0.8.4.jar:/opt/obs/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/obs/cassandra/lib/guava-r08.jar:/opt/obs/cassandra/lib/high-scale-lib-1.1.2.jar:/opt/obs/cassandra/lib/jackson-core-asl-1.4.0.jar:/opt/obs/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/opt/obs/cassandra/lib/jamm-0.2.5.jar:/opt/obs/cassandra/lib/jline-0.9.94.jar:/opt/obs/cassandra/lib/json-simple-1.1.jar:/opt/obs/cassandra/lib/libthrift-0.6.jar:/opt/obs/cassandra/lib/log4j-1.2.16.jar:/opt/obs/cassandra/lib/servlet-api-2.5-20081211.jar:/opt/obs/cassandra/lib/slf4j-api-1.6.1.jar:/opt/obs/cassandra/lib/slf4j-log4j12-1.6.1.jar:/opt/obs/cassandra/lib/snakeyaml-1.6.jar:/opt/obs/cassandra/lib/snappy-java-1.0.3.jar
>            Reporter: MaHaiyang
>
> I have 4 cassandra nodes ,and I put about 30G data to db for every nodes . It's just
4 days before I start the cluster ,but now every 4 nodes have the same problem ,JVM heap is
full  ,and  GC take no effect ,There must be some memory leak . Jmap the memory as follow:
> Object Histogram:
> num 	  #instances	#bytes	Class description
> --------------------------------------------------------------------------
> 1:		15793606	758093088	java.nio.HeapByteBuffer
> 2:		2153811	320138208	java.lang.Object[]
> 3:		6163192	197222144	org.apache.cassandra.db.Column
> 4:		2543836	175890256	int[]
> 5:		2168816	155397192	long[]
> 6:		2078123	116374888	org.cliffc.high_scale_lib.ConcurrentAutoTable$CAT
> 7:		1847111	73884440	java.math.BigInteger
> 8:		1234243	59243664	java.util.Hashtable
> 9:		1770829	58233000	char[]
> 10:		1770627	56660064	java.lang.String
> 11:		1665886	39981264	org.apache.cassandra.db.DecoratedKey
> 12:		692706	38791536	org.cliffc.high_scale_lib.NonBlockingHashMap$CHM
> 13:		1234274	37172088	java.util.Hashtable$Entry[]
> 14:		1133541	36273312	java.net.Inet4Address
> 15:		738528	35449344	org.apache.cassandra.service.ReadCallback
> 16:		2078118	33249888	org.cliffc.high_scale_lib.Counter
> 17:		1373886	32973264	org.apache.cassandra.db.ReadResponse
> 18:		1234023	29616552	org.apache.cassandra.net.Message
> 19:		1234019	29616456	org.apache.cassandra.net.Header
> 20:		1846185	29538960	org.apache.cassandra.dht.BigIntegerToken
> 21:		891378	28524096	org.apache.cassandra.utils.ExpiringMap$CacheableObject
> 22:		692706	27708240	org.cliffc.high_scale_lib.NonBlockingHashMap
> 23:		1148252	27558048	java.util.Collections$SynchronizedSet
> 24:		541977	26014896	org.apache.cassandra.db.SliceFromReadCommand
> 25:		998001	23952024	java.util.concurrent.ConcurrentSkipListMap$Node
> 26:		928792	22291008	java.util.ArrayList
> 27:		692715	22166880	java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceFieldUpdaterImpl
> 28:		891378	21393072	org.apache.cassandra.net.CallbackInfo
> 29:		1148247	18371952	java.util.Hashtable$KeySet
> 30:		731859	17564616	org.apache.cassandra.db.Row
> 31:		529991	16959712	org.apache.cassandra.db.ArrayBackedSortedColumns
> 32:		691425	16594200	org.apache.cassandra.db.AbstractColumnContainer$DeletionInfo
> 33:		648580	15565920	org.apache.cassandra.db.filter.QueryPath
> 34:		648338	15560112	org.apache.cassandra.service.RowDigestResolver
> 35:		971376	15542016	java.util.concurrent.atomic.AtomicInteger
> 36:		837418	13398688	org.apache.cassandra.utils.SimpleCondition
> 37:		535614	12854736	org.apache.cassandra.db.ColumnFamily
> 38:		725634	11610144	java.util.concurrent.atomic.AtomicReference
> 39:		195117	9365616	org.apache.cassandra.db.ThreadSafeSortedColumns
> 40:		281921	9021472	java.util.concurrent.ConcurrentSkipListMap$HeadIndex
> 41:		277679	8885728	java.util.concurrent.locks.ReentrantLock$NonfairSync
> 42:		314424	7546176	java.util.concurrent.ConcurrentSkipListMap$Index
> 43:		275186	6604464	java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
> 44:		270280	6486720	java.util.concurrent.LinkedBlockingQueue$Node
> 45:		219553	5269272	org.apache.cassandra.io.sstable.IndexSummary$KeyPosition
> 46:		106436	5108928	java.util.TreeMap
> 47:		122185	4887400	org.apache.cassandra.db.ExpiringColumn
> 48:		189968	4559232	org.apache.cassandra.db.SuperColumn
> 49:		275659	4410544	java.util.concurrent.locks.ReentrantLock
> 50:		90213	4330224	java.util.concurrent.LinkedBlockingQueue
> 51:		107026	4281040	java.util.TreeMap$Entry
> 52:		30501	4222056	* ConstMethodKlass

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message