Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 7 Jun 2016 08:33:20 +0000 (UTC)
From: "Takeshi Yamamuro (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.12973229.1464303985000.11953.1465288400989@Atlassian.JIRA>
In-Reply-To: <JIRA.12973229.1464303985000@Atlassian.JIRA>
References: <JIRA.12973229.1464303985000@Atlassian.JIRA> <JIRA.12973229.1464303985927@arcas>
Subject: [jira] [Commented] (SPARK-15585) Don't use null in data source
 options to indicate default value
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 07 Jun 2016 08:33:22 -0000


    [ https://issues.apache.org/jira/browse/SPARK-15585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15318132#comment-15318132 ] 

Takeshi Yamamuro commented on SPARK-15585:
------------------------------------------

btw,  the behavior of `df.option("sep", "|").csv("...")` changes before and after my pr #13372 applied?
`CSVOptions#getChar` does not seem to affect the behavior as follows;
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala#L67
Manually checked in spark-shell, but I got the same result between them.
Anything I missed there?


> Don't use null in data source options to indicate default value
> ---------------------------------------------------------------
>
>                 Key: SPARK-15585
>                 URL: https://issues.apache.org/jira/browse/SPARK-15585
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Reynold Xin
>            Priority: Critical
>
> See email: http://apache-spark-developers-list.1001551.n3.nabble.com/changed-behavior-for-csv-datasource-and-quoting-in-spark-2-0-0-SNAPSHOT-td17704.html
> We'd need to change DataFrameReader/DataFrameWriter in Python's csv/json/parquet/... functions to put the actual default option values as function parameters, rather than setting them to None. We can then in CSVOptions.getChar (and JSONOptions, etc) to actually return null if the value is null, rather  than setting it to default value.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org