hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jean-Daniel Cryans (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-9642) AM ZK Workers stuck doing 100% CPU on HashMap.put
Date Tue, 24 Sep 2013 01:08:03 GMT
Jean-Daniel Cryans created HBASE-9642:
-----------------------------------------

             Summary: AM ZK Workers stuck doing 100% CPU on HashMap.put
                 Key: HBASE-9642
                 URL: https://issues.apache.org/jira/browse/HBASE-9642
             Project: HBase
          Issue Type: Bug
    Affects Versions: 0.96.0
            Reporter: Jean-Daniel Cryans
            Priority: Blocker
             Fix For: 0.98.0, 0.96.0


I just noticed on my test cluster that my master is using all my CPUs even though it's completely
idle. 5 threads are doing this:

{noformat}
"AM.ZK.Worker-pool2-t34" daemon prio=10 tid=0x00007f68ac176800 nid=0x5251 runnable [0x00007f688cc83000]
   java.lang.Thread.State: RUNNABLE
	at java.util.HashMap.put(HashMap.java:374)
	at org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:954)
	at org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1419)
	at org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1247)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
{noformat}

Looking at the code, I see HBASE-9095 introduced two HashMaps *for tests only* but they end
up being used concurrently in the AM _and_ are never cleaned up. It seems to me that any master
running since that patch was committed has a time bomb in it.

I'm marking this as a blocker. [~devaraj] and [~jxiang], you guys wanna take a look at this?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message