spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lantao Jin (JIRA)" <>
Subject [jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system
Date Thu, 08 Jun 2017 18:20:18 GMT


Lantao Jin commented on SPARK-21023:

it may break existing applications
I really know the risk and hope to do the right thing. Need find a way to keep current behavior
and can easily control the behavior follow we want.

> Ignore to load default properties file is not a good choice from the perspective of system
> ------------------------------------------------------------------------------------------
>                 Key: SPARK-21023
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Submit
>    Affects Versions: 2.1.1
>            Reporter: Lantao Jin
>            Priority: Minor
> The default properties file {{spark-defaults.conf}} shouldn't be ignore to load even
though the submit arg {{--properties-file}} is set. The reasons are very easy to see:
> * Infrastructure team need continually update the {{spark-defaults.conf}} when they want
set something as default for entire cluster as a tuning purpose.
> * Application developer only want to override the parameters they really want rather
than others they even doesn't know (Set by infrastructure team).
> * The purpose of using {{\-\-properties-file}} from most of application developers is
to avoid setting dozens of {{--conf k=v}}. But if {{spark-defaults.conf}} is ignored, the
behaviour becomes unexpected finally.
> All this caused by below codes:
> {code}
>   private Properties loadPropertiesFile() throws IOException {
>     Properties props = new Properties();
>     File propsFile;
>     if (propertiesFile != null) {
>     // default conf property file will not be loaded when app developer use --properties-file
as a submit args
>       propsFile = new File(propertiesFile);
>       checkArgument(propsFile.isFile(), "Invalid properties file '%s'.", propertiesFile);
>     } else {
>       propsFile = new File(getConfDir(), DEFAULT_PROPERTIES_FILE);
>     }
>     //...
>     return props;
>   }
> {code}
> I can offer a patch to fix it if you think it make sense.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message