hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Hofhansl (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-7729) TestCatalogTrackerOnCluster.testbadOriginalRootLocation fails occasionally
Date Thu, 31 Jan 2013 21:55:12 GMT

    [ https://issues.apache.org/jira/browse/HBASE-7729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13568154#comment-13568154
] 

Lars Hofhansl commented on HBASE-7729:
--------------------------------------

I see the following event:
{{2013-01-31 04:51:22,352 INFO  [pool-1-thread-1] util.JVMClusterUtil(262): Shutdown of 1
master(s) and 1 regionserver(s) complete}}

Followed by:
{code}
013-01-31 04:51:22,354 DEBUG [Master:0;hemera.apache.org,40481,1359607875904] catalog.CatalogTracker(227):
Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@13c66d8
2013-01-31 04:51:22,354 INFO  [hemera.apache.org,40481,1359607875904.timeoutMonitor] hbase.Chore(80):
hemera.apache.org,40481,1359607875904.timeoutMonitor exiting
2013-01-31 04:51:22,354 INFO  [hemera.apache.org,40481,1359607875904.timerUpdater] hbase.Chore(80):
hemera.apache.org,40481,1359607875904.timerUpdater exiting
2013-01-31 04:51:22,355 DEBUG [Master:0;hemera.apache.org,40481,1359607875904-EventThread]
zookeeper.ZKUtil(419): hconnection-0x13c8ef1c6650002 /hbase/master does not exist. Watcher
is set.
2013-01-31 04:51:22,356 INFO  [Master:0;hemera.apache.org,40481,1359607875904] master.HMaster(410):
HMaster main thread exiting
{code}

So JVMClusterUtil concluded that is done with shutting down the cluster, which is strange,
since JVMClusterUtil loops until Thread.isAlive() returns false for all region server and
master threads.

Maybe it's a logging anomaly. In other runs I've seen as much as 8ms between cluster thinking
it's down and the Master actually leaving its main loop (the ZK threads live on much longer
anyway).
                
> TestCatalogTrackerOnCluster.testbadOriginalRootLocation fails occasionally
> --------------------------------------------------------------------------
>
>                 Key: HBASE-7729
>                 URL: https://issues.apache.org/jira/browse/HBASE-7729
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>
> Failure:
> {code}
> java.io.IOException: Shutting down
> 	at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:223)
> 	at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:86)
> 	at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:77)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:650)
> 	at org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster.testBadOriginalRootLocation(TestCatalogTrackerOnCluster.java:68)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> 	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> 	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47)
> 	at org.junit.rules.RunRules.evaluate(RunRules.java:18)
> 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> 	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> 	at org.junit.runners.Suite.runChild(Suite.java:128)
> 	at org.junit.runners.Suite.runChild(Suite.java:24)
> 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.RuntimeException: Master not initialized after 200 seconds
> 	at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:206)
> 	at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:420)
> 	at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:216)
> 	... 32 more
> {code}
> Likely caused by this:
> {code}
> 2013-01-31 04:52:23,064 FATAL [Master:0;hemera.apache.org,52696,1359607882775] master.HMaster(1493):
Unhandled exception. Starting shutdown.
> org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the
failed servers list: example.org/192.0.43.10:1234
> 	at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:425)
> 	at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1124)
> 	at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:974)
> 	at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
> 	at $Proxy19.getProtocolVersion(Unknown Source)
> 	at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
> 	at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1335)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1291)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1278)
> 	at org.apache.hadoop.hbase.catalog.CatalogTracker.getCachedConnection(CatalogTracker.java:506)
> 	at org.apache.hadoop.hbase.catalog.CatalogTracker.getRootServerConnection(CatalogTracker.java:343)
> 	at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRootServerConnection(CatalogTracker.java:327)
> 	at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRootRegionLocation(CatalogTracker.java:599)
> 	at org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:659)
> 	at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:560)
> 	at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:376)
> 	at java.lang.Thread.run(Thread.java:662)
> 2013-01-31 04:52:23,064 INFO  [Master:0;hemera.apache.org,52696,1359607882775] master.HMaster(1652):
Aborting
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message