spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-11474) Options to jdbc load are lower cased
Date Wed, 04 Nov 2015 19:05:27 GMT

     [ https://issues.apache.org/jira/browse/SPARK-11474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Apache Spark reassigned SPARK-11474:
------------------------------------

    Assignee: Apache Spark

> Options to jdbc load are lower cased
> ------------------------------------
>
>                 Key: SPARK-11474
>                 URL: https://issues.apache.org/jira/browse/SPARK-11474
>             Project: Spark
>          Issue Type: Bug
>          Components: Input/Output
>    Affects Versions: 1.5.1
>         Environment: Linux & Mac
>            Reporter: Stephen Samuel
>            Assignee: Apache Spark
>            Priority: Minor
>
> We recently upgraded from spark 1.3.0 to 1.5.1 and one of the features we wanted to take
advantage of was the fetchSize added to the jdbc data frame reader.
> In 1.5.1 there appears to be a bug or regression, whereby an options map has its keys
lowercased. This means the existing properties from prior to 1.4 are ok, such as dbtable,
url and driver, but the newer fetchSize gets converted to fetchsize.
> To re-produce:
> val conf = new SparkConf(true).setMaster("local").setAppName("fetchtest")
> val sc = new SparkContext(conf)
> val sql = new SQLContext(sc)
> val options = Map("url" -> ...., "driver" -> ...., "fetchSize" -> ....)
> val df = sql.load("jdbc", options)
> Breakpoint at line 371 in JDBCRDD and you'll see the options are all lowercased, so:
> val fetchSize = properties.getProperty("fetchSize", "0").toInt
> results in 0
> Now I know sql.load is deprecated, but this might be occuring on other methods too. The
workaround is to use the java.util.Properties overload, which keeps the case sensitive keys.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message