hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hari Sekhon (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-7594) Hive JDBC client: "out of sequence response" on large long running query
Date Fri, 01 Aug 2014 14:20:38 GMT

     [ https://issues.apache.org/jira/browse/HIVE-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Hari Sekhon updated HIVE-7594:
------------------------------

    Description: 
When executing a long running query in a JDBC client (Squirrel) to HiveServer2 after several
minutes I get this error in the client:
{code}
Error: org.apache.thrift.TApplicationException: ExecuteStatement failed: out of sequence response
SQLState:  08S01
ErrorCode: 0
{code}
I've seen this before in, iirc when running 2 queries in 1 session but I've closed the client
and run only this single query in a new session each time. I did a search and saw HIVE-6893
referring to a Metastore exception which I have in some older logs but not corresponding /
recent in these recent instances, the error seems different in this case but may be related.

The query to reproduce is "select count(*) from myTable" where myTable is a 1TB table of 620
million rows. This happens in both MR and Tez execution engines running on Yarn.

Here are all the jars I've added to the classpath (taken from Hortonworks doc http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1-latest/bk_dataintegration/content/ch_using-hive-2.html,
plus added hadoop-common, hive-exec and slf4j-api to solve class not found issues on top of
that):

commons-codec-1.4.jar
commons-logging-1.1.3.jar
hadoop-common-2.4.0.2.1.3.0-563.jar
hive-exec-0.13.0.2.1.3.0-563.jar
hive-jdbc-0.13.0.2.1.3.0-563.jar
hive-service-0.13.0.2.1.3.0-563.jar
httpclient-4.2.5.jar
httpcore-4.2.5.jar
libthrift-0.9.0.jar
slf4j-api-1.7.5.jar

I am seeing errors like this in the hiveserver2.log:
{code}
2014-08-01 15:04:31,358 ERROR [pool-5-thread-3]: server.TThreadPoolServer (TThreadPoolServer.java:run(215))
- Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:189)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
        at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
        at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
        at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
        ... 4 more

...
2014-08-01 15:06:31,520 ERROR [pool-5-thread-3]: server.TThreadPoolServer (TThreadPoolServer.java:run(215))
- Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:189)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
        at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
        at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
        at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
        ... 4 more

...

2014-08-01 15:05:41,096 ERROR [pool-4-thread-22]: operation.Operation (SQLOperation.java:run(202))
- Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Illegal Operation state transition
        at org.apache.hive.service.cli.OperationState.validateTransition(OperationState.java:94)
        at org.apache.hive.service.cli.OperationState.validateTransition(OperationState.java:99)
        at org.apache.hive.service.cli.operation.Operation.setState(Operation.java:109)
        at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:158)
        at org.apache.hive.service.cli.operation.SQLOperation.access$000(SQLOperation.java:68)
        at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:199)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
        at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:500)
        at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:208)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
{code}

Regards,

Hari Sekhon
http://www.linkedin.com/in/harisekhon

  was:
When executing a long running query in a JDBC client (Squirrel) to HiveServer2 after several
minutes I get this error in the client:
{code}
Error: org.apache.thrift.TApplicationException: ExecuteStatement failed: out of sequence response
SQLState:  08S01
ErrorCode: 0
{code}
I've seen this before in, iirc when running 2 queries in 1 session but I've closed the client
and run only this single query in a new session each time. I did a quick search of other Hive
Jiras and saw the Metastore exception jira, which I have in some older logs but not corresponding
/ recent in these recent instances.

The query to reproduce is "select count(*) from myTable" where myTable is a 1TB table of 620
million rows. This happens in both MR and Tez execution engines running on Yarn.

Here are all the jars I've added to the classpath (taken from Hortonworks doc http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1-latest/bk_dataintegration/content/ch_using-hive-2.html,
plus added hadoop-common, hive-exec and slf4j-api to solve class not found issues on top of
that):

commons-codec-1.4.jar
commons-logging-1.1.3.jar
hadoop-common-2.4.0.2.1.3.0-563.jar
hive-exec-0.13.0.2.1.3.0-563.jar
hive-jdbc-0.13.0.2.1.3.0-563.jar
hive-service-0.13.0.2.1.3.0-563.jar
httpclient-4.2.5.jar
httpcore-4.2.5.jar
libthrift-0.9.0.jar
slf4j-api-1.7.5.jar

I am seeing errors like this in the hiveserver2.log:
{code}
2014-08-01 15:04:31,358 ERROR [pool-5-thread-3]: server.TThreadPoolServer (TThreadPoolServer.java:run(215))
- Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:189)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
        at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
        at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
        at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
        ... 4 more

...
2014-08-01 15:06:31,520 ERROR [pool-5-thread-3]: server.TThreadPoolServer (TThreadPoolServer.java:run(215))
- Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:189)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
        at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
        at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
        at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
        ... 4 more

...

2014-08-01 15:05:41,096 ERROR [pool-4-thread-22]: operation.Operation (SQLOperation.java:run(202))
- Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Illegal Operation state transition
        at org.apache.hive.service.cli.OperationState.validateTransition(OperationState.java:94)
        at org.apache.hive.service.cli.OperationState.validateTransition(OperationState.java:99)
        at org.apache.hive.service.cli.operation.Operation.setState(Operation.java:109)
        at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:158)
        at org.apache.hive.service.cli.operation.SQLOperation.access$000(SQLOperation.java:68)
        at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:199)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
        at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:500)
        at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:208)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
{code}

Regards,

Hari Sekhon
http://www.linkedin.com/in/harisekhon


> Hive JDBC client: "out of sequence response" on large long running query
> ------------------------------------------------------------------------
>
>                 Key: HIVE-7594
>                 URL: https://issues.apache.org/jira/browse/HIVE-7594
>             Project: Hive
>          Issue Type: Bug
>          Components: Clients, HiveServer2
>    Affects Versions: 0.13.0
>         Environment: HDP2.1
>            Reporter: Hari Sekhon
>
> When executing a long running query in a JDBC client (Squirrel) to HiveServer2 after
several minutes I get this error in the client:
> {code}
> Error: org.apache.thrift.TApplicationException: ExecuteStatement failed: out of sequence
response
> SQLState:  08S01
> ErrorCode: 0
> {code}
> I've seen this before in, iirc when running 2 queries in 1 session but I've closed the
client and run only this single query in a new session each time. I did a search and saw HIVE-6893
referring to a Metastore exception which I have in some older logs but not corresponding /
recent in these recent instances, the error seems different in this case but may be related.
> The query to reproduce is "select count(*) from myTable" where myTable is a 1TB table
of 620 million rows. This happens in both MR and Tez execution engines running on Yarn.
> Here are all the jars I've added to the classpath (taken from Hortonworks doc http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1-latest/bk_dataintegration/content/ch_using-hive-2.html,
plus added hadoop-common, hive-exec and slf4j-api to solve class not found issues on top of
that):
> commons-codec-1.4.jar
> commons-logging-1.1.3.jar
> hadoop-common-2.4.0.2.1.3.0-563.jar
> hive-exec-0.13.0.2.1.3.0-563.jar
> hive-jdbc-0.13.0.2.1.3.0-563.jar
> hive-service-0.13.0.2.1.3.0-563.jar
> httpclient-4.2.5.jar
> httpcore-4.2.5.jar
> libthrift-0.9.0.jar
> slf4j-api-1.7.5.jar
> I am seeing errors like this in the hiveserver2.log:
> {code}
> 2014-08-01 15:04:31,358 ERROR [pool-5-thread-3]: server.TThreadPoolServer (TThreadPoolServer.java:run(215))
- Error occurred during processing of message.
> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
>         at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>         at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:189)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.thrift.transport.TTransportException
>         at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>         at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>         at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
>         at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>         at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>         at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>         at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>         ... 4 more
> ...
> 2014-08-01 15:06:31,520 ERROR [pool-5-thread-3]: server.TThreadPoolServer (TThreadPoolServer.java:run(215))
- Error occurred during processing of message.
> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
>         at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>         at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:189)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.thrift.transport.TTransportException
>         at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>         at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>         at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
>         at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>         at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>         at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>         at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>         ... 4 more
> ...
> 2014-08-01 15:05:41,096 ERROR [pool-4-thread-22]: operation.Operation (SQLOperation.java:run(202))
- Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Illegal Operation state transition
>         at org.apache.hive.service.cli.OperationState.validateTransition(OperationState.java:94)
>         at org.apache.hive.service.cli.OperationState.validateTransition(OperationState.java:99)
>         at org.apache.hive.service.cli.operation.Operation.setState(Operation.java:109)
>         at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:158)
>         at org.apache.hive.service.cli.operation.SQLOperation.access$000(SQLOperation.java:68)
>         at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:199)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
>         at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:500)
>         at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:208)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> {code}
> Regards,
> Hari Sekhon
> http://www.linkedin.com/in/harisekhon



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message