spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yin Huai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-10588) Saving a DataFrame containing only nulls to JSON doesn't work
Date Mon, 14 Sep 2015 16:21:46 GMT

    [ https://issues.apache.org/jira/browse/SPARK-10588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743770#comment-14743770
] 

Yin Huai commented on SPARK-10588:
----------------------------------

This is an expected behavior. When we write a row out, we skip those null values, which is
pretty useful to save space when writing sparse data to json.

One possible way to address this issue is to write null values only for the first row generated
by a writer.

> Saving a DataFrame containing only nulls to JSON doesn't work
> -------------------------------------------------------------
>
>                 Key: SPARK-10588
>                 URL: https://issues.apache.org/jira/browse/SPARK-10588
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.5.0
>            Reporter: Cheng Lian
>
> Snippets to reproduce this issue:
> {noformat}
> val path = "file:///tmp/spark/null"
> // A single row containing a single null double, saving to JSON, wrong
> sqlContext.
>   range(1).selectExpr("CAST(NULL AS DOUBLE) AS c0").
>   write.mode("overwrite").json(path)
> sqlContext.read.json(path).show()
> ++
> ||
> ++
> ||
> ++
> // Two rows each containing a single null double, saving to JSON, wrong
> sqlContext.
>   range(2).selectExpr("CAST(NULL AS DOUBLE) AS c0").
>   write.mode("overwrite").json(path)
> sqlContext.read.json(path).show()
> ++
> ||
> ++
> ||
> ||
> ++
> // A single row containing two null doubles, saving to JSON, wrong
> sqlContext.
>   range(1).selectExpr("CAST(NULL AS DOUBLE) AS c0", "CAST(NULL AS DOUBLE) AS c1").
>   write.mode("overwrite").json(path)
> sqlContext.read.json(path).show()
> ++
> ||
> ++
> ||
> ++
> // A single row containing a single null double, saving to Parquet, OK
> sqlContext.
>   range(1).selectExpr("CAST(NULL AS DOUBLE) AS c0").
>   write.mode("overwrite").parquet(path)
> sqlContext.read.parquet(path).show()
> +----+
> |   d|
> +----+
> |null|
> +----+
> // Two rows, one containing a single null double, one containing non-null double, saving
to JSON, OK
> sqlContext.
>   range(2).selectExpr("IF(id % 2 = 0, CAST(NULL AS DOUBLE), id) AS c0").
>   write.mode("overwrite").json(path)
> sqlContext.read.json(path).show()
> +----+
> |   d|
> +----+
> |null|
> | 1.0|
> +----+
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message