hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-9642) AM ZK Workers stuck doing 100% CPU on HashMap.put
Date Wed, 25 Sep 2013 00:08:04 GMT

    [ https://issues.apache.org/jira/browse/HBASE-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776967#comment-13776967
] 

Hudson commented on HBASE-9642:
-------------------------------

SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #759 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/759/])
HBASE-9642. AM ZK Workers stuck doing 100% CPU on HashMap.put (ddas: rev 1526009)
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMaster.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestZKBasedOpenCloseRegion.java

                
> AM ZK Workers stuck doing 100% CPU on HashMap.put
> -------------------------------------------------
>
>                 Key: HBASE-9642
>                 URL: https://issues.apache.org/jira/browse/HBASE-9642
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.96.0
>            Reporter: Jean-Daniel Cryans
>            Assignee: Devaraj Das
>            Priority: Blocker
>             Fix For: 0.98.0, 0.96.0
>
>         Attachments: 9642-1.txt, 9642-2.txt
>
>
> I just noticed on my test cluster that my master is using all my CPUs even though it's
completely idle. 5 threads are doing this:
> {noformat}
> "AM.ZK.Worker-pool2-t34" daemon prio=10 tid=0x00007f68ac176800 nid=0x5251 runnable [0x00007f688cc83000]
>    java.lang.Thread.State: RUNNABLE
> 	at java.util.HashMap.put(HashMap.java:374)
> 	at org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:954)
> 	at org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1419)
> 	at org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1247)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> {noformat}
> Looking at the code, I see HBASE-9095 introduced two HashMaps *for tests only* but they
end up being used concurrently in the AM _and_ are never cleaned up. It seems to me that any
master running since that patch was committed has a time bomb in it.
> I'm marking this as a blocker. [~devaraj] and [~jxiang], you guys wanna take a look at
this?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message