spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheng Lian (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-10177) Parquet support interprets timestamp values differently from Hive 0.14.0+
Date Mon, 24 Aug 2015 08:36:45 GMT

     [ https://issues.apache.org/jira/browse/SPARK-10177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Cheng Lian updated SPARK-10177:
-------------------------------
    Description: 
Running the following SQL under Hive 0.14.0+ (tested against 0.14.0 and 1.2.1):
{code:sql}
CREATE TABLE ts_test STORED AS PARQUET
AS SELECT CAST("2015-01-01 00:00:00" AS TIMESTAMP);
{code}
Then read the Parquet file generated by Hive with Spark SQL:
{noformat}
scala> sqlContext.read.parquet("hdfs://localhost:9000/user/hive/warehouse_hive14/ts_test").collect()
res1: Array[org.apache.spark.sql.Row] = Array([2015-01-01 12:00:00.0])
{noformat}

This issue can be easily reproduced with this test case in PR

Spark 1.4.1 works as expected in this case.

  was:
Running the following SQL under Hive 0.14.0+ (tested against 0.14.0 and 1.2.1):
{code:sql}
CREATE TABLE ts_test STORED AS PARQUET
AS SELECT CAST("2015-01-01 00:00:00" AS TIMESTAMP);
{code}
Then read the Parquet file generated by Hive with Spark SQL:
{noformat}
scala> sqlContext.read.parquet("hdfs://localhost:9000/user/hive/warehouse_hive14/ts_test").collect()
res1: Array[org.apache.spark.sql.Row] = Array([2015-01-01 12:00:00.0])
{noformat}
Spark 1.4.1 works as expected in this case.


> Parquet support interprets timestamp values differently from Hive 0.14.0+
> -------------------------------------------------------------------------
>
>                 Key: SPARK-10177
>                 URL: https://issues.apache.org/jira/browse/SPARK-10177
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.5.0
>            Reporter: Cheng Lian
>            Assignee: Cheng Lian
>            Priority: Blocker
>         Attachments: 000000_0
>
>
> Running the following SQL under Hive 0.14.0+ (tested against 0.14.0 and 1.2.1):
> {code:sql}
> CREATE TABLE ts_test STORED AS PARQUET
> AS SELECT CAST("2015-01-01 00:00:00" AS TIMESTAMP);
> {code}
> Then read the Parquet file generated by Hive with Spark SQL:
> {noformat}
> scala> sqlContext.read.parquet("hdfs://localhost:9000/user/hive/warehouse_hive14/ts_test").collect()
> res1: Array[org.apache.spark.sql.Row] = Array([2015-01-01 12:00:00.0])
> {noformat}
> This issue can be easily reproduced with this test case in PR
> Spark 1.4.1 works as expected in this case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message