hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "huaxiang sun (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16293) TestSnapshotFromMaster#testSnapshotHFileArchiving flakey
Date Wed, 27 Jul 2016 21:57:20 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15396482#comment-15396482
] 

huaxiang sun commented on HBASE-16293:
--------------------------------------

Spent some time internally with Matteo, we found the issue. 
The issue is that compaction is disabled for the table. hbase.hstore.blockingStoreFiles is
12, and the test tries to create 20 store files. When 12 is reached, it tries to compact but
compaction is disabled. Then the memstore size is exceeded and region server starts to return
RegionTooBusyException to the client. The client will retry and stuck forever. Looking into
the logic of the unitest, it does not assume how many store files it needs to create. 

It is related with HBASE-16235.

2016-07-27 13:06:18,272 INFO  [hconnection-0x5b178b4b-shared-pool15-t22] client.AsyncProcess$AsyncRequestFutureImpl(1266):
#26, table=test, attempt=13/36 failed=4468ops, last exception: org.apache.hadoop.hbase.RegionTooBusyException:
org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, regionName=test,,1469649897929.2e3e92fa2512073f502609defa3845f8.,
server=172.16.1.53,53409,1469649884198, memstoreSize=1363248, blockingMemStoreSize=100000
at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:3719)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2945)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2896)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:878)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:840)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2359)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:38447)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2212)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:118)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:189)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:169)
 on 172.16.1.53,53409,1469649884198, tracking started null, retrying after=20180ms, replay=4468ops

> TestSnapshotFromMaster#testSnapshotHFileArchiving flakey
> --------------------------------------------------------
>
>                 Key: HBASE-16293
>                 URL: https://issues.apache.org/jira/browse/HBASE-16293
>             Project: HBase
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 2.0.0
>            Reporter: huaxiang sun
>            Assignee: huaxiang sun
>
> Got the following stack trace for this failure, not sure if it is related with HBASE-9072
> -------------------------------------------------------
>  T E S T S
> -------------------------------------------------------
> Running org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster
> Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 336.042 sec <<<
FAILURE! - in org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster
> testSnapshotHFileArchiving(org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster)
 Time elapsed: 303.771 sec  <<< ERROR!
> org.junit.runners.model.TestTimedOutException: test timed out after 300000 milliseconds
> 	at java.lang.Object.wait(Native Method)
> 	at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1810)
> 	at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1784)
> 	at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1860)
> 	at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:241)
> 	at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191)
> 	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:979)
> 	at org.apache.hadoop.hbase.client.HTable.put(HTable.java:576)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:2002)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:1979)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:1967)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:1945)
> 	at org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:297)
> Results :
> Tests in error: 
>   TestSnapshotFromMaster.testSnapshotHFileArchiving:297->Object.wait:-2 ยป TestTimedOut
> Tests run: 4, Failures: 0, Errors: 1, Skipped: 0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message