Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 27 May 2016 05:28:12 +0000 (UTC)
From: "Shivaram Venkataraman (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.12973229.1464303985000.312643.1464326892928@Atlassian.JIRA>
In-Reply-To: <JIRA.12973229.1464303985000@Atlassian.JIRA>
References: <JIRA.12973229.1464303985000@Atlassian.JIRA> <JIRA.12973229.1464303985927@arcas>
Subject: [jira] [Commented] (SPARK-15585) Don't use null in data source
 options to indicate default value
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 27 May 2016 05:28:15 -0000


    [ https://issues.apache.org/jira/browse/SPARK-15585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15303523#comment-15303523 ] 

Shivaram Venkataraman commented on SPARK-15585:
-----------------------------------------------

I am not sure i completely understand the question - The way the options get passed in R [1]  is that we create a hash map and fill it in with anything passed in by the user. `NULL` is a restricted keyword in R (note that its in all caps), and it gets deserialized / passed as `null` to Scala.

[1] https://github.com/apache/spark/blob/c82883239eadc4615a3aba907cd4633cb7aed26e/R/pkg/R/SQLContext.R#L658

> Don't use null in data source options to indicate default value
> ---------------------------------------------------------------
>
>                 Key: SPARK-15585
>                 URL: https://issues.apache.org/jira/browse/SPARK-15585
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Reynold Xin
>            Priority: Critical
>
> See email: http://apache-spark-developers-list.1001551.n3.nabble.com/changed-behavior-for-csv-datasource-and-quoting-in-spark-2-0-0-SNAPSHOT-td17704.html
> We'd need to change DataFrameReader/DataFrameWriter in Python's csv/json/parquet/... functions to put the actual default option values as function parameters, rather than setting them to None. We can then in CSVOptions.getChar (and JSONOptions, etc) to actually return null if the value is null, rather  than setting it to default value.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org