spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wenchen Fan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-22387) propagate session configs to data source read/write options
Date Sun, 29 Oct 2017 13:04:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-22387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Wenchen Fan updated SPARK-22387:
--------------------------------
    Description: 
This is an open discussion. The general idea is we should allow users to set some common configs
in session conf so that they don't need to type them again and again for each data source
operations.

Proposal 1:
propagate every session config which starts with {{spark.datasource.config.}} to data source
options. The downside is, users may only want to set some common configs for a specific data
source.

Proposal 2:
propagate session config which starts with {{spark.datasource.config.myDataSource.}} only
to {{myDataSource}} operations. One downside is, some data source may not have a short name
and makes the config key pretty long, e.g. {{spark.datasource.config.com.company.foo.bar.key1}}.

Proposal 3:
Introduce a trait `WithSessionConfig` which defines session config key prefix. Then we can
pick session configs with this key-prefix and propagate it to this particular data source.

One another thing also worth to think: sometimes it's really awful if users have a type in
the config key and spent 

  was:
This is an open discussion. The general idea is we should allow users to set some common configs
in session conf so that they don't need to type them again and again for each data source
operations.

Proposal 1:
propagate every session config which starts with {{spark.datasource.config.}} to data source
options. The downside is, users may only want to set some common configs for a specific data
source.

Proposal 2:
propagate session config which starts with {{spark.datasource.config.myDataSource.}} only
to {{myDataSource}} operations. One downside is, some data source may not have a short name
and makes the config key pretty long, e.g. {{spark.datasource.config.com.company.foo.bar.key1}}.

Proposal 3:
Introduce a trait `WithSessionConfig` which defines session config key prefix. Then we can
pick session configs with this key-prefix and propagate it to this particular data sourcde.


> propagate session configs to data source read/write options
> -----------------------------------------------------------
>
>                 Key: SPARK-22387
>                 URL: https://issues.apache.org/jira/browse/SPARK-22387
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 2.3.0
>            Reporter: Wenchen Fan
>
> This is an open discussion. The general idea is we should allow users to set some common
configs in session conf so that they don't need to type them again and again for each data
source operations.
> Proposal 1:
> propagate every session config which starts with {{spark.datasource.config.}} to data
source options. The downside is, users may only want to set some common configs for a specific
data source.
> Proposal 2:
> propagate session config which starts with {{spark.datasource.config.myDataSource.}}
only to {{myDataSource}} operations. One downside is, some data source may not have a short
name and makes the config key pretty long, e.g. {{spark.datasource.config.com.company.foo.bar.key1}}.
> Proposal 3:
> Introduce a trait `WithSessionConfig` which defines session config key prefix. Then we
can pick session configs with this key-prefix and propagate it to this particular data source.
> One another thing also worth to think: sometimes it's really awful if users have a type
in the config key and spent 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message