cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benedict (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-8860) Too many java.util.HashMap$Entry objects in heap
Date Thu, 26 Feb 2015 14:37:04 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14338466#comment-14338466
] 

Benedict commented on CASSANDRA-8860:
-------------------------------------

{quote}
"CompactionExecutor:14" daemon prio=10 tid=0x00002aaace7a7800 nid=0x7e31 runnable [0x0000000042a79000]
   java.lang.Thread.State: RUNNABLE
	at java.util.HashMap.transfer(HashMap.java:605)
	at java.util.HashMap.resize(HashMap.java:585)
	at java.util.HashMap.addEntry(HashMap.java:883)
	at java.util.HashMap.put(HashMap.java:509)
	at java.util.HashSet.add(HashSet.java:217)
	at org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.createOverlapChain(SizeTieredCompactionStrategy.java:235)
	at org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.filterColdSSTables(SizeTieredCompactionStrategy.java:202)
	at org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.getNextBackgroundSSTables(SizeTieredCompactionStrategy.java:85)
	at org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.getNextBackgroundTask(SizeTieredCompactionStrategy.java:320)
	- locked <0x000000064f254560> (a org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy)
	at org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:72)
	- locked <0x000000064f254500> (a org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
	at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:234)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
{quote}

Looks like a viable candidate, and was introduced recently. I can't see anything fundamentally
broken about it, although it does have O(n^2) memory utilisation in the number of sstables.
This would require around 8500 sstables to be involved in the compaction to create the current
heap pressure, which is both: 1) possible for some users (so probably we need to rethink);
and 2) also pretty uncommon? so perhaps not the issue here?

Could we get a list of the data files present on disk for this table, your log file, and perhaps
a heap dump?

> Too many java.util.HashMap$Entry objects in heap
> ------------------------------------------------
>
>                 Key: CASSANDRA-8860
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8860
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Cassandra 2.1.3, jdk 1.7u51
>            Reporter: Phil Yang
>         Attachments: cassandra-env.sh, cassandra.yaml, jmap.txt, jstack.txt
>
>
> While I upgrading my cluster to 2.1.3, I find some nodes (not all) may have GC issue
after the node restarting successfully. Old gen grows very fast and most of the space can
not be recycled after setting its status to normal immediately. The qps of both reading and
writing are very low and there is no heavy compaction.
> Jmap result seems strange that there are too many java.util.HashMap$Entry objects in
heap, where in my experience the "[B" is usually the No1.
> If I downgrade it to 2.1.1, this issue will not appear.
> I uploaded conf files and jstack/jmap outputs. I'll upload heap dump if someone need
it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message