hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aihua Xu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-14162) Allow disabling of long running job on Hive On Spark On YARN
Date Tue, 15 Nov 2016 15:07:59 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15667393#comment-15667393
] 

Aihua Xu commented on HIVE-14162:
---------------------------------

[~xuefuz] My first patch doesn't work.  I'm thinking of creating a new JobContextImpl (so
new SparkContext) for each new job in RemoteDriver rather than sharing for this jira. After
the job is done, the SparkContext should be dropped. Of course, this will be controlled by
a configuration. This implementation will have performance impact since they are not sharing
resource now, but seems it's good for releasing the resource. Also maybe help to diagnostics
since the executions and the logs are separate. Do you see any issues with that approach?




> Allow disabling of long running job on Hive On Spark On YARN
> ------------------------------------------------------------
>
>                 Key: HIVE-14162
>                 URL: https://issues.apache.org/jira/browse/HIVE-14162
>             Project: Hive
>          Issue Type: New Feature
>          Components: Spark
>            Reporter: Thomas Scott
>            Assignee: Aihua Xu
>            Priority: Minor
>         Attachments: HIVE-14162.1.patch
>
>
> Hive On Spark launches a long running process on the first query to handle all queries
for that user session. In some use cases this is not desired, for instance when using Hue
with large intervals between query executions.
> Could we have a property that would cause long running spark jobs to be terminated after
each query execution and started again for the next one?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message