hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jean-Daniel Cryans (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-9642) AM ZK Workers stuck doing 100% CPU on HashMap.put
Date Tue, 24 Sep 2013 21:14:04 GMT

    [ https://issues.apache.org/jira/browse/HBASE-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776754#comment-13776754

Jean-Daniel Cryans commented on HBASE-9642:

I'm doing some more testing on RC3 and I keep hitting this bug, thanks for the fix Dev :)
> AM ZK Workers stuck doing 100% CPU on HashMap.put
> -------------------------------------------------
>                 Key: HBASE-9642
>                 URL: https://issues.apache.org/jira/browse/HBASE-9642
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.96.0
>            Reporter: Jean-Daniel Cryans
>            Assignee: Devaraj Das
>            Priority: Blocker
>             Fix For: 0.98.0, 0.96.0
>         Attachments: 9642-1.txt, 9642-2.txt
> I just noticed on my test cluster that my master is using all my CPUs even though it's
completely idle. 5 threads are doing this:
> {noformat}
> "AM.ZK.Worker-pool2-t34" daemon prio=10 tid=0x00007f68ac176800 nid=0x5251 runnable [0x00007f688cc83000]
>    java.lang.Thread.State: RUNNABLE
> 	at java.util.HashMap.put(HashMap.java:374)
> 	at org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:954)
> 	at org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1419)
> 	at org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1247)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> {noformat}
> Looking at the code, I see HBASE-9095 introduced two HashMaps *for tests only* but they
end up being used concurrently in the AM _and_ are never cleaned up. It seems to me that any
master running since that patch was committed has a time bomb in it.
> I'm marking this as a blocker. [~devaraj] and [~jxiang], you guys wanna take a look at

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message