phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Flavio Pompermaier (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (PHOENIX-4523) phoenix.schema.isNamespaceMappingEnabled problem
Date Wed, 10 Jan 2018 08:31:01 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-4523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16319888#comment-16319888
] 

Flavio Pompermaier edited comment on PHOENIX-4523 at 1/10/18 8:30 AM:
----------------------------------------------------------------------

Hi ~karanmehta93,
You can find the complete stacktrace at the end of the comment.
Now I respond to your questions:

1. I'm using Phoenix in a Flink (1.3.1)  job. There's no problem until the connection is opened
(almost) simultaneously in the cluster. After the error, the Flink job fails and I find a
row in the SYSTEM:MUTEX (UPGRADE_MUTEX_UNLOCKED). This does not happen during local test from
the IDE, it happens only when you try to run the job on a cluster, where multiple hosts try
to create that table and put a row inside it (IMHO)
2.  I don't know if there's an easy way to debug it locally because I can reproduce it only
when I submit the Flink job to the cluster. I think here there are 2 different errors/problems:

# first, there should not be attempt to create the mutex table (and this happens because getSystemTableNames()
doesn't handle properly the case where namespace are enabled and the migration has been completed)
# second, it seems that TableExistsException is not catched within createSysMutexTable() because
is wrapped by a org.apache.hadoop.ipc.RemoteException, and here I don't know why that exception
is not wrapped during local debug (connected to a remote cluster as well..)

3. I'm using 4.13.1-HBase-1.2 on client side, and the Cloudera parcel (for CDH 5.11.2) on
the server side
4. The error is thrown on a system where the SYSTEM tables are already migrated (if you refer
to SYSTEM. => SYSTEM: migration)

Thanks,
Flavio

{code:java}
java.lang.IllegalArgumentException: open() failed.org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableExistsException):
SYSTEM:MUTEX
        at org.apache.flink.api.java.io.jdbc.JDBCInputFormat.openInputFormat(JDBCInputFormat.java:144)
        at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:115)
        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableExistsException):
SYSTEM:MUTEX
        at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:2492)
        at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:2384)
        at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:76)
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2384)
        at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:255)
        at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:150)
        at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:270)
        at org.apache.flink.api.java.io.jdbc.JDBCInputFormat.openInputFormat(JDBCInputFormat.java:138)
        ... 3 more
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableExistsException):
SYSTEM:MUTEX
        at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.prepareCreate(CreateTableProcedure.java:288)
        at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:109)
        at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:59)
        at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119)
        at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:498)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1061)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:856)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:809)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:495)
{code}




was (Author: f.pompermaier):
Hi ~karanmehta93,
You can find the complete stacktrace at the end of the comment.
Now I respond to your questions:

1. I'm using Phoenix in a Flink (1.3.1)  job. There's no problem until the connection is opened
(almost) simultaneously in the cluster. After the error, the Flink job fails and I find a
row in the SYSTEM:MUTEX (UPGRADE_MUTEX_UNLOCKED). This does not happen during local test from
the IDE, it happens only when you try to run the job on a cluster, where multiple hosts try
to create that table and put a row inside it (IMHO)
2.  I don't know if there's an easy way to debug it locally because I can reproduce it only
when I submit the Flink job to the cluster. I think here there are 2 different errors/problems:

# first, there should not be attempt to create the mutex table (and this happens because getSystemTableNames()
doesn't handle properly the case where namespace are enabled and the migration has been completed)
# second, it seems that TableExistsException is not catched within createSysMutexTable() because
is wrapped by a org.apache.hadoop.ipc.RemoteException, and here I don't know why that exception
is not wrapped during local debug (connected to a remote cluster as well..)

3. I'm using 4.13.1-HBase-1.2 on client side, and the Cloudera parcel (for CDH 5.11.2) on
the server side
4. The error is thrown on a system where the SYSTEM tables are already migrated (if you refer
to SYSTEM. => SYSTEM: migration

Thanks,
Flavio

{code:java}
java.lang.IllegalArgumentException: open() failed.org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableExistsException):
SYSTEM:MUTEX
        at org.apache.flink.api.java.io.jdbc.JDBCInputFormat.openInputFormat(JDBCInputFormat.java:144)
        at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:115)
        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableExistsException):
SYSTEM:MUTEX
        at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:2492)
        at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:2384)
        at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:76)
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2384)
        at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:255)
        at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:150)
        at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:270)
        at org.apache.flink.api.java.io.jdbc.JDBCInputFormat.openInputFormat(JDBCInputFormat.java:138)
        ... 3 more
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableExistsException):
SYSTEM:MUTEX
        at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.prepareCreate(CreateTableProcedure.java:288)
        at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:109)
        at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:59)
        at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119)
        at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:498)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1061)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:856)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:809)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:495)
{code}



> phoenix.schema.isNamespaceMappingEnabled problem
> ------------------------------------------------
>
>                 Key: PHOENIX-4523
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-4523
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 4.13.1
>            Reporter: Flavio Pompermaier
>            Assignee: Karan Mehta
>
> I'm using Phoenix 4.13 for CDH 5.11.2 parcel and enabling schemas made my code unusable.
> I think that this is not a bug of the CDH release, but of all 4.13.x releases.
> I have many parallel Phoenix connections and I always get the following exception:
> {code:java}
> Caused by: java.sql.SQLException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableExistsException):
SYSTEM:MUTEX
> 	at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:2492)
> 	at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:2384)
> 	at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:76)
> 	at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2384)
> 	at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:255)
> 	at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:150)
> 	at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221)
> 	at java.sql.DriverManager.getConnection(DriverManager.java:664)
> 	at java.sql.DriverManager.getConnection(DriverManager.java:270)
> {code}
> This is caused by the fact that all the times the SYSTEM tables are recreated, and this
cannot be done simultaneously.
> Trying to debug the issue I found that in ConnectionQueryServicesImpl.createSysMutexTable()
the call to getSystemTableNames() always return an empty array and the SYSTEM:MUTEX  table
is always recreated.
> This because getSystemTableNames() doesn't consider the case when system tables have
namespace enabled. Right now that method tries to get all tables starting with *SYSTEM.\**,
while it should try to get the list of *SYSTEM:\** tables..
> I hope this could get fixed very soon,
> Flavio



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message