hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Yu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-8116) TestSnapshotCloneIndependence fails in trunk builds intermittently
Date Thu, 14 Mar 2013 23:28:12 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13602885#comment-13602885
] 

Ted Yu commented on HBASE-8116:
-------------------------------

>From build #3958, we can get more clue:
{code}
2013-03-14 07:05:49,273 INFO  [pool-1-thread-1] hbase.ResourceChecker(171): after: client.TestSnapshotCloneIndependence#testOfflineSnapshotRegionOperationsIndependent
Thread=276 (was 272)
Potentially hanging thread: RegionServer:0;janus.apache.org,57486,1363244621206-splits-1363244745097
	java.lang.Object.wait(Native Method)
	java.lang.Thread.join(Thread.java:1258)
	java.lang.Thread.join(Thread.java:1332)
	org.apache.hadoop.hbase.util.HasThread.join(HasThread.java:89)
	org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughters(SplitTransaction.java:378)
	org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:475)
	org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:68)
	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	java.lang.Thread.run(Thread.java:722)

Potentially hanging thread: janus.apache.org,57486,1363244621206-daughterOpener=72730ff3c3745f6be273de75948a2c30
	java.lang.Thread.sleep(Native Method)
	org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:4100)
	org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3988)
	org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
	org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
	org.apache.hadoop.hbase.regionserver.HRegionFileSystem.writeRegionInfoFileContent(HRegionFileSystem.java:401)
	org.apache.hadoop.hbase.regionserver.HRegionFileSystem.writeRegionInfoOnFilesystem(HRegionFileSystem.java:473)
	org.apache.hadoop.hbase.regionserver.HRegionFileSystem.checkRegionInfoOnFilesystem(HRegionFileSystem.java:435)
	org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:567)
	org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:546)
	org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4041)
	org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughterRegion(SplitTransaction.java:527)
	org.apache.hadoop.hbase.regionserver.SplitTransaction$DaughterOpener.run(SplitTransaction.java:508)
	java.lang.Thread.run(Thread.java:722)

Potentially hanging thread: janus.apache.org,57486,1363244621206-daughterOpener=adff955f142634edcdec1a090f47645f
	java.lang.Object.wait(Native Method)
	java.lang.Object.wait(Object.java:503)
	org.apache.hadoop.ipc.Client.call(Client.java:1093)
	org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
	$Proxy10.delete(Unknown Source)
	sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:601)
	org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
	org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
	$Proxy10.delete(Unknown Source)
	sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:601)
	org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:267)
	$Proxy19.delete(Unknown Source)
	org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:981)
	org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:245)
	org.apache.hadoop.fs.FilterFileSystem.delete(FilterFileSystem.java:154)
	org.apache.hadoop.hbase.util.FSUtils.deleteDirectory(FSUtils.java:166)
	org.apache.hadoop.hbase.regionserver.HRegionFileSystem.cleanupTempDir(HRegionFileSystem.java:119)
	org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:571)
	org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:546)
	org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4041)
	org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughterRegion(SplitTransaction.java:527)
	org.apache.hadoop.hbase.regionserver.SplitTransaction$DaughterOpener.run(SplitTransaction.java:508)
{code}
Looks like there might be race between the two daughterOpener threads.
                
> TestSnapshotCloneIndependence fails in trunk builds intermittently
> ------------------------------------------------------------------
>
>                 Key: HBASE-8116
>                 URL: https://issues.apache.org/jira/browse/HBASE-8116
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Ted Yu
>
> I was looking at https://builds.apache.org/job/HBase-TRUNK/3959/testReport/org.apache.hadoop.hbase.client/TestSnapshotCloneIndependence/testOfflineSnapshotRegionOperationsIndependent/
and found the following:
> {code}
> 2013-03-14 11:11:07,323 INFO  [pool-1-thread-1] hbase.ResourceChecker(171): after: client.TestSnapshotCloneIndependence#testOfflineSnapshotRegionOperationsIndependent
Thread=275 (was 273)
> Potentially hanging thread: janus.apache.org,53542,1363259346619-daughterOpener=34172719c055b187015add70302ab50b
> 	java.lang.Object.wait(Native Method)
> 	java.lang.Object.wait(Object.java:503)
> 	org.apache.hadoop.ipc.Client.call(Client.java:1093)
> 	org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> 	$Proxy10.rename(Unknown Source)
> 	sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> 	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	java.lang.reflect.Method.invoke(Method.java:601)
> 	org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
> 	org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
> 	$Proxy10.rename(Unknown Source)
> 	sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> 	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	java.lang.reflect.Method.invoke(Method.java:601)
> 	org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:267)
> 	$Proxy19.rename(Unknown Source)
> 	org.apache.hadoop.hdfs.DFSClient.rename(DFSClient.java:955)
> 	org.apache.hadoop.hdfs.DistributedFileSystem.rename(DistributedFileSystem.java:227)
> 	org.apache.hadoop.fs.FilterFileSystem.rename(FilterFileSystem.java:144)
> 	org.apache.hadoop.hbase.regionserver.HRegionFileSystem.writeRegionInfoOnFilesystem(HRegionFileSystem.java:476)
> 	org.apache.hadoop.hbase.regionserver.HRegionFileSystem.checkRegionInfoOnFilesystem(HRegionFileSystem.java:435)
> 	org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:567)
> 	org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:546)
> 	org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4041)
> 	org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughterRegion(SplitTransaction.java:527)
> 	org.apache.hadoop.hbase.regionserver.SplitTransaction$DaughterOpener.run(SplitTransaction.java:508)
> 	java.lang.Thread.run(Thread.java:722)
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message