Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 726E7115D7 for ; Thu, 10 Apr 2014 17:20:25 +0000 (UTC) Received: (qmail 35164 invoked by uid 500); 10 Apr 2014 17:20:24 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 34918 invoked by uid 500); 10 Apr 2014 17:20:24 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 34774 invoked by uid 99); 10 Apr 2014 17:20:22 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Apr 2014 17:20:22 +0000 Date: Thu, 10 Apr 2014 17:20:22 +0000 (UTC) From: "Jimmy Xiang (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HBASE-10897) On master start, deadlock if refresh UI MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-10897: -------------------------------- Status: Patch Available (was: Open) > On master start, deadlock if refresh UI > --------------------------------------- > > Key: HBASE-10897 > URL: https://issues.apache.org/jira/browse/HBASE-10897 > Project: HBase > Issue Type: Bug > Affects Versions: 0.99.0 > Reporter: stack > Assignee: Jimmy Xiang > Fix For: 0.99.0 > > Attachments: hbase-10897.patch, hbase-10897_v2.patch, hbase-10897_v3.patch, hbase-10897_v4.patch > > > Playing w/ MTTR recovery on trunk, master starting up deadlocked: > Waiting to finish active master initialization: > {code} > "ActiveMasterManager" daemon prio=10 tid=0x00007fafb5dc3800 nid=0x5fb5 waiting for monitor entry [0x00007faf8f57d000] > java.lang.Thread.State: BLOCKED (on object monitor) > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683) > - waiting to lock <0x000000064ab4b9a8> (a java.lang.Object) > at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53) > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029) > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989) > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830) > at org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305) > at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77) > at org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118) > at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:101) > at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264) > at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169) > at org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:164) > at org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:107) > at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:766) > at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:539) > at org.apache.hadoop.hbase.catalog.MetaReader.fullScanOfMeta(MetaReader.java:140) > at org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.isMetaTableUpdated(MetaMigrationConvertingToPB.java:164) > at org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.updateMetaIfNecessary(MetaMigrationConvertingToPB.java:131) > at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:567) > at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:147) > at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1242) > at java.lang.Thread.run(Thread.java:744) > {code} > ... but the master servlet has the lock while trying to access master: > {code} > "686004346@qtp-2101021459-0" daemon prio=10 tid=0x00007fafb5d2a800 nid=0x5fb1 waiting on condition [0x00007faf8f87f000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1562) > - locked <0x000000064ab4b9a8> (a java.lang.Object) > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1597) > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1805) > - locked <0x000000064ab4b9a8> (a java.lang.Object) > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.listTables(ConnectionManager.java:2481) > at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:321) > at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__userTables(MasterStatusTmplImpl.java:530) > at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:255) > at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:382) > at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:372) > at org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:102) > ... > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)