spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Reynold Xin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-15585) Don't use null in data source options to indicate default value
Date Mon, 06 Jun 2016 07:44:56 GMT

    [ https://issues.apache.org/jira/browse/SPARK-15585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15316288#comment-15316288
] 

Reynold Xin commented on SPARK-15585:
-------------------------------------

[~maropu] I think the best way is to advise users to pass \u0000 in. Can you check if that
is possible in both Python/Scala?


> Don't use null in data source options to indicate default value
> ---------------------------------------------------------------
>
>                 Key: SPARK-15585
>                 URL: https://issues.apache.org/jira/browse/SPARK-15585
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Reynold Xin
>            Priority: Critical
>
> See email: http://apache-spark-developers-list.1001551.n3.nabble.com/changed-behavior-for-csv-datasource-and-quoting-in-spark-2-0-0-SNAPSHOT-td17704.html
> We'd need to change DataFrameReader/DataFrameWriter in Python's csv/json/parquet/...
functions to put the actual default option values as function parameters, rather than setting
them to None. We can then in CSVOptions.getChar (and JSONOptions, etc) to actually return
null if the value is null, rather  than setting it to default value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message