hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chengxiang Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-7330) Create SparkTask
Date Fri, 01 Aug 2014 06:30:43 GMT

    [ https://issues.apache.org/jira/browse/HIVE-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14081977#comment-14081977
] 

Chengxiang Li commented on HIVE-7330:
-------------------------------------

[~chinnalalam] I agree with you that we could refresh local resource, monitoring job outside
SparkClient, As a wrapper class of SparkContext, SparkClient would be just in charge of submit
spark job. One concerns is that, in your patch, spark job is submitted in SparkTask instead
of SparkClient, and SparkContext is exposed and used in SparkTask. For better responsibility
division, should we just use SparkClient to submit spark job and load environment information
in SparkTask? what do you think?

> Create SparkTask
> ----------------
>
>                 Key: HIVE-7330
>                 URL: https://issues.apache.org/jira/browse/HIVE-7330
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Xuefu Zhang
>            Assignee: Chinna Rao Lalam
>         Attachments: HIVE-7330-spark.patch, HIVE-7330.1-spark.patch
>
>
> SparkTask handles the execution of SparkWork. It will execute a graph of map and reduce
work using a SparkClient instance.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message