hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuefu Zhang (JIRA)" <>
Subject [jira] [Commented] (HIVE-7436) Load Spark configuration into Hive driver
Date Tue, 22 Jul 2014 04:34:38 GMT


Xuefu Zhang commented on HIVE-7436:

[~chengxiang li] I didn't find hive on tez is taking configuration from tez-site.xml. The
main configurations I saw is in, which are named more Hive configuration. I
assume those can be put in hive-site.xml or set in command line. 

    HIVE_SERVER2_TEZ_DEFAULT_QUEUES("hive.server2.tez.default.queues", ""),

Would you mind pointing me to the place where tez-site.xml is processed?

Having a controlled set of configurations in Hive's terms has the benefits of clarity and
simplicity. If we read spark default conf or allow user to specify any spark configuration
in hive-site, it's probably a nightmare for documentation or explaining strange behaviours
because of user setting a configure that hive never thought of.

Nevertheless, I'm open to processing spark default conf if Tez also does so, though personally
I prefer a control over the set of spark configurations.

You're right, SparkContext should be per user session as multi-tenant is supported in hive.
So this means that we can not support changing spark configurations through hive cli set command.
I think this is fine. User can set these configurations in command line (if admin allows it),
but once SparkContext is created upon the first query, resetting those will not have any effect
for subsequent queries until user disconnects and reconnects. This can be documented.

> Load Spark configuration into Hive driver
> -----------------------------------------
>                 Key: HIVE-7436
>                 URL:
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Chengxiang Li
>            Assignee: Chengxiang Li
>         Attachments: HIVE-7436-Spark.1.patch, HIVE-7436-Spark.2.patch
> load Spark configuration into Hive driver, there are 3 ways to setup spark configurations:
> #  Configure properties in spark configuration file(spark-defaults.conf).
> #  Java property.
> #  System environment.
> Spark support configuration through system environment just for compatible with previous
scripts, we won't support in Hive on Spark. Hive on Spark load defaults from java properties,
then load properties from configuration file, and override existed properties.
> configuration steps:
> # Create spark-defaults.conf, and place it in the /etc/spark/conf configuration directory.
>     please refer to [] for configuration
of spark-defaults.conf.
> # Create the $SPARK_CONF_DIR environment variable and set it to the location of spark-defaults.conf.
>     export SPARK_CONF_DIR=/etc/spark/conf
> # Add $SAPRK_CONF_DIR to the $HADOOP_CLASSPATH environment variable.
> NO PRECOMMIT TESTS. This is for spark-branch only.

This message was sent by Atlassian JIRA

View raw message