spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Takeshi Yamamuro (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-15585) Don't use null in data source options to indicate default value
Date Tue, 07 Jun 2016 08:33:20 GMT

    [ https://issues.apache.org/jira/browse/SPARK-15585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15318132#comment-15318132
] 

Takeshi Yamamuro commented on SPARK-15585:
------------------------------------------

btw,  the behavior of `df.option("sep", "|").csv("...")` changes before and after my pr #13372
applied?
`CSVOptions#getChar` does not seem to affect the behavior as follows;
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala#L67
Manually checked in spark-shell, but I got the same result between them.
Anything I missed there?


> Don't use null in data source options to indicate default value
> ---------------------------------------------------------------
>
>                 Key: SPARK-15585
>                 URL: https://issues.apache.org/jira/browse/SPARK-15585
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Reynold Xin
>            Priority: Critical
>
> See email: http://apache-spark-developers-list.1001551.n3.nabble.com/changed-behavior-for-csv-datasource-and-quoting-in-spark-2-0-0-SNAPSHOT-td17704.html
> We'd need to change DataFrameReader/DataFrameWriter in Python's csv/json/parquet/...
functions to put the actual default option values as function parameters, rather than setting
them to None. We can then in CSVOptions.getChar (and JSONOptions, etc) to actually return
null if the value is null, rather  than setting it to default value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message