phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Francis Chuang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-4374) Flakyness with Phoenix 4.13.0 and HBase 1.3.1: RuntimeException: org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.NamespaceNotFoundException:
Date Mon, 20 Nov 2017 10:21:01 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16259055#comment-16259055
] 

Francis Chuang commented on PHOENIX-4374:
-----------------------------------------

I am currently planning to produce a dump of all the protobuf messages exchanged between the
app and the query server. This should allow replaying the requests against the server to see
what this is happening.

The code is already in place and I just need to trigger the error again.

> Flakyness with Phoenix 4.13.0 and HBase 1.3.1:  RuntimeException: org.apache.phoenix.exception.PhoenixIOException:
org.apache.hadoop.hbase.NamespaceNotFoundException:
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: PHOENIX-4374
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-4374
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 4.13.0
>            Reporter: Francis Chuang
>         Attachments: hbase--master-m9edd51-phoenix.m9edd51.log, root-queryserver.log,
tephra-service--m9edd51-phoenix.m9edd51.log
>
>
> I am using the Phoenix Query Server via my [Go Avatica SQL driver|https://github.com/Boostport/avatica].
> In terms of my set up I am running Phoenix 4.13.0 and HBase 1.3.1 in docker with a single
node HBase using local storage. The dockerfile is available here: https://github.com/Boostport/hbase-phoenix-all-in-one
> Today, I updated one of my projects to use the latest version of the above image (Phoenix
4.13.0 and HBase 1.3.1) and my integration tests against Phoenix + HBase have become extremely
flaky. The tests use a mix of transactional and non-transactional tables.
> The flakyness is that random tests will fail with the same error. If I rerun the tests,
they sometimes pass and sometimes fail, and it is not clear why this is happening.
> In all of these tests, I am doing the following:
> 1. Create the schema.
> 2. Create tables.
> 3. Insert, delete and read data.
> 4. Delete the tables and schema.
> This is the error I get when trying to drop the schema:
> {code:java}
> An error was encountered while processing your request: RuntimeException: org.apache.phoenix.exception.PhoenixIOException:
org.apache.hadoop.hbase.NamespaceNotFoundException: INITTEST
>                         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
>                         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>                         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>                         at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>                         at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
>                         at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
>                         at org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
>                         at org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114)
>                         at org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85)
>                         at org.apache.hadoop.hbase.master.HMaster$15.run(HMaster.java:2717)
>                         at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133)
>                         at org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2705)
>                         at org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:496)
>                         at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58601)
>                         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
>                         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
>                         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
>                         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
>                 Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.NamespaceNotFoundException):
INITTEST
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.prepareDelete(DeleteNamespaceProcedure.java:243)
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:83)
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:49)
>                         at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139)
>                         at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1148)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:943)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:896)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:78)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:498)
>                  -> PhoenixIOException: org.apache.hadoop.hbase.NamespaceNotFoundException:
INITTEST
>                         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
>                         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>                         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>                         at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>                         at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
>                         at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
>                         at org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
>                         at org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114)
>                         at org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85)
>                         at org.apache.hadoop.hbase.master.HMaster$15.run(HMaster.java:2717)
>                         at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133)
>                         at org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2705)
>                         at org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:496)
>                         at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58601)
>                         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
>                         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
>                         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
>                         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
>                 Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.NamespaceNotFoundException):
INITTEST
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.prepareDelete(DeleteNamespaceProcedure.java:243)
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:83)
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:49)
>                         at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139)
>                         at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1148)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:943)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:896)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:78)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:498)
>                  -> NamespaceNotFoundException: org.apache.hadoop.hbase.NamespaceNotFoundException:
INITTEST
>                         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
>                         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>                         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>                         at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>                         at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
>                         at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
>                         at org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
>                         at org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114)
>                         at org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85)
>                         at org.apache.hadoop.hbase.master.HMaster$15.run(HMaster.java:2717)
>                         at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133)
>                         at org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2705)
>                         at org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:496)
>                         at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58601)
>                         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
>                         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
>                         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
>                         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
>                 Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.NamespaceNotFoundException):
INITTEST
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.prepareDelete(DeleteNamespaceProcedure.java:243)
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:83)
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:49)
>                         at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139)
>                         at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1148)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:943)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:896)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:78)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:498)
>                  -> RemoteWithExtrasException: org.apache.hadoop.hbase.NamespaceNotFoundException:
INITTEST
>                         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
>                         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>                         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>                         at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>                         at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
>                         at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
>                         at org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
>                         at org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114)
>                         at org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85)
>                         at org.apache.hadoop.hbase.master.HMaster$15.run(HMaster.java:2717)
>                         at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133)
>                         at org.apache.hadoop.hbase.master.HMaster.deleteNamespace(HMaster.java:2705)
>                         at org.apache.hadoop.hbase.master.MasterRpcServices.deleteNamespace(MasterRpcServices.java:496)
>                         at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58601)
>                         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
>                         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
>                         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
>                         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
>                 Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.NamespaceNotFoundException):
INITTEST
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.prepareDelete(DeleteNamespaceProcedure.java:243)
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:83)
>                         at org.apache.hadoop.hbase.master.procedure.DeleteNamespaceProcedure.executeFromState(DeleteNamespaceProcedure.java:49)
>                         at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139)
>                         at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1148)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:943)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:896)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:78)
>                         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:498)
> {code}
> I have also attached the following:
> - Logs from tephra
> - Logs from PQS/Avatica
> - Logs from HBase
> Further update:
> Rolled back to the HBase 1.2 Phoenix 4.11 image and tests are rock solid.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message