hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Vary (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-16451) Race condition between HiveStatement.getQueryLog and HiveStatement.runAsyncOnServer
Date Fri, 21 Apr 2017 08:57:04 GMT

    [ https://issues.apache.org/jira/browse/HIVE-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978339#comment-15978339
] 

Peter Vary commented on HIVE-16451:
-----------------------------------

Thanks [~vihangk1] for taking the time to look at this!

Currently during the {{HiveStatement.runAsyncOnServer}} the {{isQueryClosed}} flag is flipping:
- Constructor sets initially sets it to true
- The call {{closeClientOperation}} sets it to false
- The call {{initFlags}} sets it to true again

This happens even on the happy path, when there is no cancellation or errors, and could generate
the "Method getQueryLog() failed. The statement has been closed or cancelled." warning message.

The getQueryLog, and runAsyncOnServer currently used parallel in very limited situations,
which might not lead to race conditions in other cases. At least I have to find any concrete
example, so I concentrated on the current problem.

Changing the HiveStatement to a thread safe one would be a big change. I could do that, but
was not sure that the community would agree with that - Shall we do that [~vihangk1], [~ychena]?
Would it worth the performance impact? Might worth a letter to the dev list....

Thanks,
Peter

 

> Race condition between HiveStatement.getQueryLog and HiveStatement.runAsyncOnServer
> -----------------------------------------------------------------------------------
>
>                 Key: HIVE-16451
>                 URL: https://issues.apache.org/jira/browse/HIVE-16451
>             Project: Hive
>          Issue Type: Bug
>          Components: Beeline, JDBC
>    Affects Versions: 3.0.0
>            Reporter: Peter Vary
>            Assignee: Peter Vary
>         Attachments: HIVE-16451.02.patch, HIVE-16451.03.patch, HIVE-16451.patch
>
>
> During the BeeLineDriver testing I have met the following race condition:
> - Run the query asynchronously through BeeLine
> - Querying the logs in the BeeLine
> In the following code:
> {code:title=HiveStatement.runAsyncOnServer}
>   private void runAsyncOnServer(String sql) throws SQLException {
>     checkConnection("execute");
>     closeClientOperation();
>     initFlags();
> [..]
>   }
> {code}
> {code:title=HiveStatement.getQueryLog}
>   public List<String> getQueryLog(boolean incremental, int fetchSize)
>       throws SQLException, ClosedOrCancelledStatementException {
> [..]
>     try {
>       if (stmtHandle != null) {
> [..]
>       } else {
>         if (isQueryClosed) {
>           throw new ClosedOrCancelledStatementException("Method getQueryLog() failed.
The " +
>               "statement has been closed or cancelled.");
>         } else {
>           return logs;
>         }
>       }
>     } catch (SQLException e) {
> [..]
>     }
> [..]
>   }
> {code}
> The runAsyncOnServer {{closeClientOperation}} sets {{isQueryClosed}} flag to true:
> {code:title=HiveStatement.closeClientOperation}
>   void closeClientOperation() throws SQLException {
> [..]
>     isQueryClosed = true;
>     isExecuteStatementFailed = false;
>     stmtHandle = null;
>   }
> {code}
> The {{initFlags}} sets it to false:
> {code}
>   private void initFlags() {
>     isCancelled = false;
>     isQueryClosed = false;
>     isLogBeingGenerated = true;
>     isExecuteStatementFailed = false;
>     isOperationComplete = false;
>   }
> {code}
> If the {{getQueryLog}} is called after the {{closeClientOperation}}, but before the {{initFlags}},
then we will have a following warning if verbose mode is set to true in BeeLine:
> {code}
> Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method getQueryLog()
failed. The statement has been closed or cancelled. (state=,code=0)
> {code}
> This caused this fail:
> https://builds.apache.org/job/PreCommit-HIVE-Build/4691/testReport/org.apache.hadoop.hive.cli/TestBeeLineDriver/testCliDriver_smb_mapjoin_11_/
> {code}
> Error Message
> Client result comparison failed with error code = 1 while executing fname=smb_mapjoin_11
> 16a17
> > Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method getQueryLog()
failed. The statement has been closed or cancelled. (state=,code=0)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message