hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-9642) AM ZK Workers stuck doing 100% CPU on HashMap.put
Date Tue, 24 Sep 2013 19:37:04 GMT

    [ https://issues.apache.org/jira/browse/HBASE-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776664#comment-13776664
] 

Hadoop QA commented on HBASE-9642:
----------------------------------

{color:green}+1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12604839/9642-2.txt
  against trunk revision .

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 6 new or modified
tests.

    {color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 1.0 profile.

    {color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 2.0 profile.

    {color:green}+1 javadoc{color}.  The javadoc tool did not generate any warning messages.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version
1.3.9) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:green}+1 lineLengths{color}.  The patch does not introduce lines longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

    {color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7356//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7356//console

This message is automatically generated.
                
> AM ZK Workers stuck doing 100% CPU on HashMap.put
> -------------------------------------------------
>
>                 Key: HBASE-9642
>                 URL: https://issues.apache.org/jira/browse/HBASE-9642
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.96.0
>            Reporter: Jean-Daniel Cryans
>            Assignee: Devaraj Das
>            Priority: Blocker
>             Fix For: 0.98.0, 0.96.0
>
>         Attachments: 9642-1.txt, 9642-2.txt
>
>
> I just noticed on my test cluster that my master is using all my CPUs even though it's
completely idle. 5 threads are doing this:
> {noformat}
> "AM.ZK.Worker-pool2-t34" daemon prio=10 tid=0x00007f68ac176800 nid=0x5251 runnable [0x00007f688cc83000]
>    java.lang.Thread.State: RUNNABLE
> 	at java.util.HashMap.put(HashMap.java:374)
> 	at org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:954)
> 	at org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1419)
> 	at org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1247)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> {noformat}
> Looking at the code, I see HBASE-9095 introduced two HashMaps *for tests only* but they
end up being used concurrently in the AM _and_ are never cleaned up. It seems to me that any
master running since that patch was committed has a time bomb in it.
> I'm marking this as a blocker. [~devaraj] and [~jxiang], you guys wanna take a look at
this?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message