spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-23604) ParquetInteroperabilityTest timestamp test should use Statistics.hasNonNullValue
Date Mon, 05 Mar 2018 21:08:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-23604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386769#comment-16386769
] 

Apache Spark commented on SPARK-23604:
--------------------------------------

User 'henryr' has created a pull request for this issue:
https://github.com/apache/spark/pull/20740

> ParquetInteroperabilityTest timestamp test should use Statistics.hasNonNullValue
> --------------------------------------------------------------------------------
>
>                 Key: SPARK-23604
>                 URL: https://issues.apache.org/jira/browse/SPARK-23604
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 2.3.0
>            Reporter: Henry Robinson
>            Priority: Minor
>
> We ran into an issue with a downstream build of Spark running against a custom Parquet
build where {{ParquetInteroperabilityTestSuite}} started failing because {{Statistics.isEmpty}}
changed its behavior as of PARQUET-1217. ({{isEmpty() now considers whether there are 0 or
more nulls, and by default {{num_nulls}} is 0 for 'empty' stats objects).
> The test really cares about whether the statistics object has values, so a very simple
fix to use {{hasNonNullValue}} instead corrects the issue. Filing it now because it's a backwards-compatible
fix to the current Parquet version so we can fix it right now before we hit the issue in the
future. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message