phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Taylor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-3485) CSVBulkLoadToolIT failing consistently because transaction service client init fails
Date Wed, 16 Nov 2016 02:14:58 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669095#comment-15669095
] 

James Taylor commented on PHOENIX-3485:
---------------------------------------

LGTM. Thanks, [~samarthjain]. Please review too, [~rajeshbabu] and make sure we're not making
any invalid assumptions here.

> CSVBulkLoadToolIT failing consistently because transaction service client init fails
> ------------------------------------------------------------------------------------
>
>                 Key: PHOENIX-3485
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-3485
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: Samarth Jain
>            Assignee: Samarth Jain
>         Attachments: PHOENIX-3485.patch
>
>
> Stuck threads
> Client side:
> Thread 1
> {code}
> "main" prio=5 tid=0x00007fbcc1800000 nid=0x1903 waiting on condition [0x0000000102f5e000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000007abe8f720> (a java.util.concurrent.FutureTask)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425)
> 	at java.util.concurrent.FutureTask.get(FutureTask.java:187)
> 	at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1659)
> 	at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1616)
> 	at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1276)
> 	at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1256)
> 	at org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:1441)
> 	at org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2275)
> 	at org.apache.phoenix.schema.MetaDataClient.createIndex(MetaDataClient.java:1428)
> 	at org.apache.phoenix.compile.CreateIndexCompiler$1.execute(CreateIndexCompiler.java:85)
> 	at org.apache.phoenix.jdbc.PhoenixStatement$3.call(PhoenixStatement.java:355)
> 	at org.apache.phoenix.jdbc.PhoenixStatement$3.call(PhoenixStatement.java:1)
> 	at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> 	at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:337)
> 	at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1459)
> 	at org.apache.phoenix.end2end.CsvBulkLoadToolIT.testImportOneIndexTable(CsvBulkLoadToolIT.java:309)
> 	at org.apache.phoenix.end2end.CsvBulkLoadToolIT.testImportOneLocalIndexTable(CsvBulkLoadToolIT.java:297)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> 	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> 	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> 	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> 	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> 	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> {code}
> Client side Thread 2 
> {code}
> "hconnection-0x480a20e3-shared--pool18-t9" daemon prio=5 tid=0x00007fbcc61b5800 nid=0x1f50b
in Object.wait() [0x00000001212c2000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <0x00000007ada86798> (a org.apache.hadoop.hbase.ipc.RpcClient$Call)
> 	at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1484)
> 	- locked <0x00000007ada86798> (a org.apache.hadoop.hbase.ipc.RpcClient$Call)
> 	at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1691)
> 	at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1750)
> 	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:31660)
> 	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1621)
> 	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:93)
> 	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:90)
> 	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:115)
> 	- locked <0x00000007ada86130> (a org.apache.hadoop.hbase.client.RpcRetryingCaller)
> 	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:91)
> 	- locked <0x00000007ada86130> (a org.apache.hadoop.hbase.client.RpcRetryingCaller)
> 	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:96)
> 	at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:57)
> 	at org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService$Stub.createTable(MetaDataProtos.java:16499)
> 	at org.apache.phoenix.query.ConnectionQueryServicesImpl$7.call(ConnectionQueryServicesImpl.java:1458)
> 	at org.apache.phoenix.query.ConnectionQueryServicesImpl$7.call(ConnectionQueryServicesImpl.java:1)
> 	at org.apache.hadoop.hbase.client.HTable$17.call(HTable.java:1647)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> {code}
> Server side:
> {code}
> "MetadataRpcServer.handler=3,queue=0,port=56678" daemon prio=5 tid=0x00007fbcc18f0000
nid=0x15f03 waiting on condition [0x000000011c573000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000007af048a38> (a com.google.common.util.concurrent.AbstractFuture$Sync)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> 	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:280)
> 	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> 	at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:132)
> 	at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:999)
> 	at org.apache.twill.zookeeper.ForwardingZKClientService.startAndWait(ForwardingZKClientService.java:51)
> 	at org.apache.phoenix.query.ConnectionQueryServicesImpl.initTxServiceClient(ConnectionQueryServicesImpl.java:383)
> 	at org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection(ConnectionQueryServicesImpl.java:397)
> 	at org.apache.phoenix.query.ConnectionQueryServicesImpl.access$8(ConnectionQueryServicesImpl.java:390)
> 	at org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2373)
> 	- locked <0x00000007ae4c5bf8> (a org.apache.phoenix.query.ConnectionQueryServicesImpl)
> 	at org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:1)
> 	at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:76)
> 	at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2351)
> 	at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:232)
> 	at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:147)
> 	at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:202)
> 	at java.sql.DriverManager.getConnection(DriverManager.java:571)
> 	at java.sql.DriverManager.getConnection(DriverManager.java:187)
> 	at org.apache.phoenix.util.QueryUtil.getConnection(QueryUtil.java:341)
> 	at org.apache.phoenix.util.QueryUtil.getConnectionOnServer(QueryUtil.java:328)
> 	at org.apache.phoenix.util.QueryUtil.getConnectionOnServer(QueryUtil.java:318)
> 	at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.createTable(MetaDataEndpointImpl.java:1489)
> 	at org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService.callMethod(MetaDataProtos.java:16282)
> 	at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:6041)
> 	at org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3520)
> 	at org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3502)
> 	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31194)
> 	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2149)
> 	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104)
> 	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> 	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> 	at java.lang.Thread.run(Thread.java:745)
> {code}
> On the server side, the MetadataEndPointImpl#createTable thread is stuck trying to initiate
the transaction service client.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message