hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuefu Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-12046) Re-create spark client if connection is dropped
Date Wed, 07 Oct 2015 12:47:26 GMT

    [ https://issues.apache.org/jira/browse/HIVE-12046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946785#comment-14946785
] 

Xuefu Zhang commented on HIVE-12046:
------------------------------------

Look at the patch again and it seems that it handles the case where the remote spark client
is in a bad state when trying to submit a spark job. This is good. However, it's unclear that
what's going to happen when either getDefaultParallelism() or getExecutorCount() is called
in such a situation. Also even in case that exectue() is called, the remote client can become
bad right after the isActive() check.

Therefore, I think we need to define a scope for this JIRA. If we want to be resilient to
the connection loss, then we need to consider more cases and the way to handle it. However,
it's also acceptable in my opinion that we can detect the error and ask user to log out and
log in again to get a valid session. The latter seems simpler and easier.

> Re-create spark client if connection is dropped
> -----------------------------------------------
>
>                 Key: HIVE-12046
>                 URL: https://issues.apache.org/jira/browse/HIVE-12046
>             Project: Hive
>          Issue Type: Bug
>          Components: Spark
>            Reporter: Jimmy Xiang
>            Assignee: Jimmy Xiang
>            Priority: Minor
>             Fix For: 1.3.0, 2.0.0
>
>         Attachments: HIVE-12046.1.patch
>
>
> Currently, if the connection to the spark cluster is dropped, the spark client will stay
in a bad state. A new Hive session is needed to re-establish the connection. It is better
to auto reconnect in this case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message