spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shengzhe Yao (JIRA)" <>
Subject [jira] [Resolved] (SPARK-1048) Create spark-site.xml or spark-site.yaml for configuration
Date Tue, 13 May 2014 21:23:17 GMT


Shengzhe Yao resolved SPARK-1048.

    Resolution: Duplicate

Patrick submit a patch which introduces spark-defaults.conf to set system properties. This
seems to achieve our original purpose: allow user to set configuration values in a well-known
config file and Spark will automatically pick it up.

commit fb98488fc8e68cc84f6e0750fd4e9e29029879d2
Author: Patrick Wendell <>
Date:   Mon Apr 21 10:26:33 2014 -0700

    Clean up and simplify Spark configuration

    Over time as we've added more deployment modes, this have gotten a bit unwieldy with user-facing
configuration options in Spark. Going forward we'll advise all users to run `spark-submit`
to launch applications. This is a WIP patch but it makes the following improvements:

    1. Improved `` which was missing a lot of things users now set in
that file.
    2. Removes the shipping of SPARK_CLASSPATH, SPARK_JAVA_OPTS, and SPARK_LIBRARY_PATH to
the executors on the cluster. This was an ugly hack. Instead it introduces config variables
spark.executor.extraJavaOpts, spark.executor.extraLibraryPath, and spark.executor.extraClassPath.
    3. Adds ability to set these same variables for the driver using `spark-submit`.
    4. Allows you to load system properties from a `spark-defaults.conf` file when running
`spark-submit`. This will allow setting both SparkConf options and other system properties
utilized by `spark-submit`.
    5. Made `SPARK_LOCAL_IP` an environment variable rather than a SparkConf property. This
is more consistent with it being set on each node.

    Author: Patrick Wendell <>

> Create spark-site.xml or spark-site.yaml for configuration
> ----------------------------------------------------------
>                 Key: SPARK-1048
>                 URL:
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>    Affects Versions: 0.8.1, 0.9.0, 1.0.0
>            Reporter: Shengzhe Yao
>            Priority: Critical
> Currently, spark doesn't provide the ability to pass configuration as a file, people
have to explicitly specify them on the command line. This becomes a bigger issue when deploying
spark on a cluster and people want to specify parameters other than default value. 
> As of now, spark is being a top apache project and we need to pay more attention to the
configuration. Most apache projects provide a xml configuration file (like hdfs, hadoop, hbase,
etc.) and probably we want to do the same thing for spark. The advantages are obvious, it
helps developers to specify their own spark configuration for the cluster and add/remove configuration
parameters will be much easier via file than via system property. 

This message was sent by Atlassian JIRA

View raw message