spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "yucai (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-25206) Wrong data may be returned for Parquet
Date Sat, 25 Aug 2018 03:34:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16592425#comment-16592425
] 

yucai edited comment on SPARK-25206 at 8/25/18 3:33 AM:
--------------------------------------------------------

[~dongjoon] , correct me if I am wrong.
{code:java}
spark.range(10).write.parquet("/tmp/data")
sql("DROP TABLE t")
sql("CREATE TABLE t (ID LONG) USING parquet LOCATION '/tmp/data'")
sql("select * from t where id > 0").show{code}
Based on 2.3.1,

Backport SPARK-25132, Spark will show nothing if pushdown enabled and show "1 to 9" if pushdown
disabled.

Backport -SPARK-25132- + SPARK-24716, Spark will pushdown nothing and it shows "1 to 9".

-SPARK-25132- +  +-SPARK-24716-+  + SPARK-25207, Spark will pushdown "id > 0" correctly
and shows "1 to 9".


was (Author: yucai):
[~dongjoon] , correct me if I am wrong.
{code:java}
spark.range(10).write.parquet("/tmp/data")
sql("DROP TABLE t")
sql("CREATE TABLE t (ID LONG) USING parquet LOCATION '/tmp/data'")
sql("select * from t where id > 0").show{code}
Based on 2.3.1,

Backport SPARK-25132, Spark will show nothing if pushdown enabled and show "1 to 9" if pushdown
disabled.

Backport -SPARK-25132- + SPARK-24716, Spark will pushdown nothing and it shows "1 to 9".

-SPARK-25132- + -SPARK-24716- + SPARK-25207, Spark will pushdown "id > 0" correctly
and shows "1 to 9".

> Wrong data may be returned for Parquet
> --------------------------------------
>
>                 Key: SPARK-25206
>                 URL: https://issues.apache.org/jira/browse/SPARK-25206
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.2, 2.3.1
>            Reporter: yucai
>            Priority: Blocker
>              Labels: correctness
>         Attachments: image-2018-08-24-18-05-23-485.png, image-2018-08-24-22-33-03-231.png,
image-2018-08-24-22-34-11-539.png, image-2018-08-24-22-46-05-346.png, image-2018-08-25-09-54-53-219.png,
image-2018-08-25-10-04-21-901.png, pr22183.png
>
>
> In current Spark 2.3.1, below query returns wrong data silently.
> {code:java}
> spark.range(10).write.parquet("/tmp/data")
> sql("DROP TABLE t")
> sql("CREATE TABLE t (ID LONG) USING parquet LOCATION '/tmp/data'")
> scala> sql("select * from t").show
> +----+
> |  ID|
> +----+
> |null|
> |null|
> |null|
> |null|
> |null|
> |null|
> |null|
> |null|
> |null|
> |null|
> +----+
> scala> sql("select * from t where id > 0").show
> +---+
> | ID|
> +---+
> +---+
> scala> sql("set spark.sql.parquet.filterPushdown").show
> +--------------------+-----+
> |                 key|value|
> +--------------------+-----+
> |spark.sql.parquet...| true|
> +--------------------+-----+
> scala> sql("set spark.sql.parquet.filterPushdown=false").show
> +--------------------+-----+
> |                 key|value|
> +--------------------+-----+
> |spark.sql.parquet...|false|
> +--------------------+-----+
> scala> sql("select * from t where id > 0").show
> +---+
> | ID|
> +---+
> +---+
> {code}
>  
> *Root Cause*
> Spark pushdowns FilterApi.gt(intColumn("{color:#ff0000}ID{color}"), 0: Integer) into
parquet, but {color:#ff0000}ID{color} does not exist in /tmp/data (parquet is case sensitive,
it has {color:#ff0000}id{color} actually).
> So no records are returned.
> In Spark 2.1, the user will get Exception:
> {code:java}
> Caused by: java.lang.IllegalArgumentException: Column [ID] was not found in schema!{code}
> But in Spark 2.3, they will get the wrong results sliently.
>  
> Since SPARK-24716, Spark uses Parquet schema instead of Hive metastore schema to do the
pushdown, perfect for this issue.
> [~yumwang], [~cloud_fan], [~smilegator], any thoughts? Should we backport it?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message