hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chengxiang Li (JIRA)" <>
Subject [jira] [Commented] (HIVE-7436) Load Spark configuration into Hive driver
Date Wed, 30 Jul 2014 07:10:38 GMT


Chengxiang Li commented on HIVE-7436:

[~xuefuz] HADOOP_CONF_DIR is added to HADOOP_CLASSPATH in, so as HIVE_CONF_DIR
in if we only load spark configuration file from classpath, there are 2 choices:
# export SPARK_CONF_DIR, and add it to HADOOP_CLASSPATH manually.
# commit a patch which would add SPARK_CONF_DIR to HADOOP_CLASSPATH in hive scripts(such as export SPARK_CONF_DIR. 

my concern about supporting load spark configuration file from SPARK_CONF_DIR in implementation
level is that:
# HADOOP/HIVE/HIVE on TEZ only load configuration file from classpath actually.
# it may introduce more complexity, like what should we do if different spark configuration
file available on SPARK_CONF_DIR and HADOOP_CLASSPATH both?

The way how to configure Hive on Tez is similar as current Hive on Spark. [Hive on Tez Configuration|]

> Load Spark configuration into Hive driver
> -----------------------------------------
>                 Key: HIVE-7436
>                 URL:
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Chengxiang Li
>            Assignee: Chengxiang Li
>             Fix For: spark-branch
>         Attachments: HIVE-7436-Spark.1.patch, HIVE-7436-Spark.2.patch, HIVE-7436-Spark.3.patch
> load Spark configuration into Hive driver, there are 3 ways to setup spark configurations:
> #  Java property.
> #  Configure properties in spark configuration file(spark-defaults.conf).
> #  Hive configuration file(hive-site.xml).
> The below configuration has more priority, and would overwrite previous configuration
with the same property name.
> Please refer to [] for all configurable
properties of spark, and you can configure spark configuration in Hive through following ways:
> # Configure through spark configuration file.
> #* Create spark-defaults.conf, and place it in the /etc/spark/conf configuration directory.
configure properties in spark-defaults.conf in java properties format.
> #* Create the $SPARK_CONF_DIR environment variable and set it to the location of spark-defaults.conf.
>     export SPARK_CONF_DIR=/etc/spark/conf
> #* Add $SAPRK_CONF_DIR to the $HADOOP_CLASSPATH environment variable.
> # Configure through hive configuration file.
> #* edit hive-site.xml in hive conf directory, configure properties in spark-defaults.conf
in xml format.
> Hive driver default spark properties:
> ||name||default value||description||
> |spark.master|local|Spark master url.|
> ||Hive on Spark|Default Spark application name.|
> NO PRECOMMIT TESTS. This is for spark-branch only.

This message was sent by Atlassian JIRA

View raw message