hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jimmy Xiang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-10897) On master start, deadlock if refresh UI
Date Wed, 09 Apr 2014 23:18:17 GMT

     [ https://issues.apache.org/jira/browse/HBASE-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jimmy Xiang updated HBASE-10897:
--------------------------------

    Attachment: hbase-10897_v3.patch

Attached v3 that shows master web UI as soon as possible. Added checking before each section
is available.

> On master start, deadlock if refresh UI
> ---------------------------------------
>
>                 Key: HBASE-10897
>                 URL: https://issues.apache.org/jira/browse/HBASE-10897
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.99.0
>            Reporter: stack
>            Assignee: Jimmy Xiang
>             Fix For: 0.99.0
>
>         Attachments: hbase-10897.patch, hbase-10897_v2.patch, hbase-10897_v3.patch
>
>
> Playing w/ MTTR recovery on trunk, master starting up deadlocked:
> Waiting to finish active master initialization:
> {code}
> "ActiveMasterManager" daemon prio=10 tid=0x00007fafb5dc3800 nid=0x5fb5 waiting for monitor
entry [0x00007faf8f57d000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683)
>         - waiting to lock <0x000000064ab4b9a8> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53)
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029)
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989)
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830)
>         at org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305)
>         at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77)
>         at org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118)
>         at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:101)
>         at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
>         at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169)
>         at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:164)
>         at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:107)
>         at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:766)
>         at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:539)
>         at org.apache.hadoop.hbase.catalog.MetaReader.fullScanOfMeta(MetaReader.java:140)
>         at org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.isMetaTableUpdated(MetaMigrationConvertingToPB.java:164)
>         at org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.updateMetaIfNecessary(MetaMigrationConvertingToPB.java:131)
>         at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:567)
>         at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:147)
>         at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1242)
>         at java.lang.Thread.run(Thread.java:744)
> {code}
> ... but the master servlet has the lock while trying to access master:
> {code}
> "686004346@qtp-2101021459-0" daemon prio=10 tid=0x00007fafb5d2a800 nid=0x5fb1 waiting
on condition [0x00007faf8f87f000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>         at java.lang.Thread.sleep(Native Method)
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1562)
>         - locked <0x000000064ab4b9a8> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1597)
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1805)
>         - locked <0x000000064ab4b9a8> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.listTables(ConnectionManager.java:2481)
>         at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:321)
>         at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__userTables(MasterStatusTmplImpl.java:530)
>         at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:255)
>         at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:382)
>         at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:372)
>         at org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:102)
> ...
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message