spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tejas Patil (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-12926) SQLContext to disallow users passing non-sql configs
Date Wed, 20 Jan 2016 14:28:39 GMT

     [ https://issues.apache.org/jira/browse/SPARK-12926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tejas Patil updated SPARK-12926:
--------------------------------
    Description: 
Users unknowingly try to set core Spark configs in sqlContext but later realise that it didn't
work.

{color:red}
scala> sqlContext.sql("SET spark.shuffle.memoryFraction=0.4")
res3: org.apache.spark.sql.DataFrame = [key: string, value: string]

scala> sqlContext.getConf.get("spark.shuffle.memoryFraction")
java.util.NoSuchElementException: spark.shuffle.memoryFraction
	at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:193)
	at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:193)
	at scala.Option.getOrElse(Option.scala:120)
	at org.apache.spark.SparkConf.get(SparkConf.scala:193)
{color}

We could do this:
- for configs starting with "spark.", allow only if it starts with "spark.sql.". Otherwise
disallow.
- allow anything else.

This will be a simple change in SqlConf :
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala#L621

  was:
Users unknowingly try to set core Spark configs in sqlContext but later realise that it didn't
work.

{color:red}
scala> sqlContext.sql("SET spark.shuffle.memoryFraction=0.4")
res3: org.apache.spark.sql.DataFrame = [key: string, value: string]

scala> sc.getConf.get("spark.shuffle.memoryFraction")
java.util.NoSuchElementException: spark.shuffle.memoryFraction
	at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:193)
	at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:193)
	at scala.Option.getOrElse(Option.scala:120)
	at org.apache.spark.SparkConf.get(SparkConf.scala:193)
{color}

We could do this:
- for configs starting with "spark.", allow only if it starts with "spark.sql.". Otherwise
disallow.
- allow anything else.

This will be a simple change in SqlConf :
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala#L621


> SQLContext to disallow users passing non-sql configs
> ----------------------------------------------------
>
>                 Key: SPARK-12926
>                 URL: https://issues.apache.org/jira/browse/SPARK-12926
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 1.6.0
>            Reporter: Tejas Patil
>            Priority: Trivial
>
> Users unknowingly try to set core Spark configs in sqlContext but later realise that
it didn't work.
> {color:red}
> scala> sqlContext.sql("SET spark.shuffle.memoryFraction=0.4")
> res3: org.apache.spark.sql.DataFrame = [key: string, value: string]
> scala> sqlContext.getConf.get("spark.shuffle.memoryFraction")
> java.util.NoSuchElementException: spark.shuffle.memoryFraction
> 	at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:193)
> 	at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:193)
> 	at scala.Option.getOrElse(Option.scala:120)
> 	at org.apache.spark.SparkConf.get(SparkConf.scala:193)
> {color}
> We could do this:
> - for configs starting with "spark.", allow only if it starts with "spark.sql.". Otherwise
disallow.
> - allow anything else.
> This will be a simple change in SqlConf :
> https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala#L621



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message