spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dongjoon Hyun (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-22267) Spark SQL incorrectly reads ORC file when column order is different
Date Wed, 15 Nov 2017 19:44:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-22267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16254038#comment-16254038
] 

Dongjoon Hyun commented on SPARK-22267:
---------------------------------------

[~mpetruska]. I made this issue during that PR.
Please see my comment at that time, https://github.com/apache/spark/pull/19470#issuecomment-336230552
.
We need more effort on this issue later. For Spark 2.3, convertMetastoreOrc with new OrcFileFormat
will be the best option for us.

> Spark SQL incorrectly reads ORC file when column order is different
> -------------------------------------------------------------------
>
>                 Key: SPARK-22267
>                 URL: https://issues.apache.org/jira/browse/SPARK-22267
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.6.3, 2.0.2, 2.1.0, 2.2.0
>            Reporter: Dongjoon Hyun
>
> For a long time, Apache Spark SQL returns incorrect results when ORC file schema is different
from metastore schema order.
> {code}
> scala> Seq(1 -> 2).toDF("c1", "c2").write.format("parquet").mode("overwrite").save("/tmp/p")
> scala> Seq(1 -> 2).toDF("c1", "c2").write.format("orc").mode("overwrite").save("/tmp/o")
> scala> sql("CREATE EXTERNAL TABLE p(c2 INT, c1 INT) STORED AS parquet LOCATION '/tmp/p'")
> scala> sql("CREATE EXTERNAL TABLE o(c2 INT, c1 INT) STORED AS orc LOCATION '/tmp/o'")
> scala> spark.table("p").show  // Parquet is good.
> +---+---+
> | c2| c1|
> +---+---+
> |  2|  1|
> +---+---+
> scala> spark.table("o").show    // This is wrong.
> +---+---+
> | c2| c1|
> +---+---+
> |  1|  2|
> +---+---+
> scala> spark.read.orc("/tmp/o").show  // This is correct.
> +---+---+
> | c1| c2|
> +---+---+
> |  1|  2|
> +---+---+
> {code}
> *TESTCASE*
> {code}
>   test("SPARK-22267 Spark SQL incorrectly reads ORC files when column order is different")
{
>     withTempDir { dir =>
>       val path = dir.getCanonicalPath
>       Seq(1 -> 2).toDF("c1", "c2").write.format("orc").mode("overwrite").save(path)
>       checkAnswer(spark.read.orc(path), Row(1, 2))
>       Seq("true", "false").foreach { value =>
>         withTable("t") {
>           withSQLConf(HiveUtils.CONVERT_METASTORE_ORC.key -> value) {
>             sql(s"CREATE EXTERNAL TABLE t(c2 INT, c1 INT) STORED AS ORC LOCATION '$path'")
>             checkAnswer(spark.table("t"), Row(2, 1))
>           }
>         }
>       }
>     }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message