spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <>
Subject [jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system
Date Thu, 08 Jun 2017 18:17:18 GMT


Marcelo Vanzin commented on SPARK-21023:

bq.  When and where the new config option be set? 

That's what makes that option awkward. It would have to be set in the user config or in the
command line with {{\-\-conf}}. So it's not that much different from a new command line option,
other than it avoids a new command line option.

> Ignore to load default properties file is not a good choice from the perspective of system
> ------------------------------------------------------------------------------------------
>                 Key: SPARK-21023
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Submit
>    Affects Versions: 2.1.1
>            Reporter: Lantao Jin
>            Priority: Minor
> The default properties file {{spark-defaults.conf}} shouldn't be ignore to load even
though the submit arg {{--properties-file}} is set. The reasons are very easy to see:
> * Infrastructure team need continually update the {{spark-defaults.conf}} when they want
set something as default for entire cluster as a tuning purpose.
> * Application developer only want to override the parameters they really want rather
than others they even doesn't know (Set by infrastructure team).
> * The purpose of using {{\-\-properties-file}} from most of application developers is
to avoid setting dozens of {{--conf k=v}}. But if {{spark-defaults.conf}} is ignored, the
behaviour becomes unexpected finally.
> All this caused by below codes:
> {code}
>   private Properties loadPropertiesFile() throws IOException {
>     Properties props = new Properties();
>     File propsFile;
>     if (propertiesFile != null) {
>     // default conf property file will not be loaded when app developer use --properties-file
as a submit args
>       propsFile = new File(propertiesFile);
>       checkArgument(propsFile.isFile(), "Invalid properties file '%s'.", propertiesFile);
>     } else {
>       propsFile = new File(getConfDir(), DEFAULT_PROPERTIES_FILE);
>     }
>     //...
>     return props;
>   }
> {code}
> I can offer a patch to fix it if you think it make sense.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message