hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Gray (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-3493) HMaster sometimes hangs during initialization due to missing notify call
Date Mon, 31 Jan 2011 20:12:29 GMT

    [ https://issues.apache.org/jira/browse/HBASE-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988870#comment-12988870
] 

Jonathan Gray commented on HBASE-3493:
--------------------------------------

+1.  i don't see a notify either

> HMaster sometimes hangs during initialization due to missing notify call
> ------------------------------------------------------------------------
>
>                 Key: HBASE-3493
>                 URL: https://issues.apache.org/jira/browse/HBASE-3493
>             Project: HBase
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 0.90.0
>         Environment: Linux, HBase 0.90
>            Reporter: Bruno Dumon
>         Attachments: assignment-manager-regions-notify-patch.txt
>
>
> During HMaster.finishInitialization(), assignRootAndMeta() is called, which at some point
does:
> {code}
> this.assignmentManager.waitForAssignment(HRegionInfo.FIRST_META_REGIONINFO);
> {code}
> And waitForAssignment does this:
> {code}
>     synchronized(regions) {
>       while(!regions.containsKey(regionInfo)) {
>         regions.wait();
>       }
>     }
> {code}
> However, I could not find any call to regions.notify(), so this could wait forever.
> I have not noticed this problem on a real cluster, only when using HBaseTestingUtility
on a slow and low-memory Hudson server (I can reproduce it on my local machine by creating
some background load). Adding a notify() call when regions.put() is called seems to fix the
problem. Will attach patch.
> For reference, this was based on seeing the following from jstack:
> {noformat}
> "Master:0;lat:44776" prio=10 tid=0x08d5b400 nid=0x381a in Object.wait() [0x9c76d000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xa5196fb8> (a java.util.TreeMap)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.hadoop.hbase.master.AssignmentManager.waitForAssignment(AssignmentManager.java:1152)
>         - locked <0xa5196fb8> (a java.util.TreeMap)
>         at org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:440)
>         at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:382)
>         at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:277)
>         at java.lang.Thread.run(Thread.java:619)
> {noformat}

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message