hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chengxiang Li (JIRA)" <>
Subject [jira] [Updated] (HIVE-8835) identify dependency scope for Remote Spark Context.[Spark Branch]
Date Tue, 18 Nov 2014 06:48:34 GMT


Chengxiang Li updated HIVE-8835:
    Attachment: HIVE-8835.1-spark.patch

Remote Spark Context need at least 3 dependencies to go through a query.
# spark assembly jar
# hive exec bundle jar
# spark-client jar

Hive ql module depends on spark-client module, we should add spark-client into hive exec bundle

> identify dependency scope for Remote Spark Context.[Spark Branch]
> -----------------------------------------------------------------
>                 Key: HIVE-8835
>                 URL:
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Chengxiang Li
>            Assignee: Chengxiang Li
>              Labels: Spark-M3
>         Attachments: HIVE-8835.1-spark.patch
> While submit job through Remote Spark Context, spark RDD graph generation and job submit
is executed in remote side, so we have to add hive  related dependency into its classpath
with spark.driver.extraClassPath. instead of add all hive/hadoop dependency, we should narrow
the scope and identify what dependency remote spark context required. 

This message was sent by Atlassian JIRA

View raw message