hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Szehon Ho" <>
Subject Re: Review Request 30337: HIVE-9482 : Hive parquet timestamp compatibility
Date Wed, 28 Jan 2015 20:10:57 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated Jan. 28, 2015, 8:10 p.m.)

Review request for hive and Brock Noland.


Address review comments.

Bugs: HIVE-9482

Repository: hive-git


In current Hive implementation, timestamps are stored in UTC (converted from current timezone),
based on original parquet timestamp spec.
However, we find this is not compatibility with other tools, and after some investigation
it is not the way of the other file formats, or even some databases (Hive Timestamp is more
equivalent of 'timestamp without timezone' datatype).

This is the first part of the fix, which will restore compatibility with parquet-timestamp
files generated by external tools by skipping conversion on reading.

Later fix will change the write path to not convert, and stop the read-conversion even for
files written by Hive itself.

Diffs (updated)

  common/src/java/org/apache/hadoop/hive/conf/ 64e7e0a 
  data/files/parquet_external_time.parq PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ a86d6f4 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ 23bb364 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ 872900b

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ 11772be

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ eeb3838

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ af28b4c 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/ 3f8e4d7

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/ 4e4d7fd

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/ c647b24 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/ 41b5f1c 
  ql/src/test/org/apache/hadoop/hive/ql/io/parquet/serde/ 2e788bd

  ql/src/test/queries/clientpositive/parquet_external_time.q PRE-CREATION 
  ql/src/test/results/clientpositive/parquet_external_time.q.out PRE-CREATION 



Added new unit tests (TestParquetTimestampUtils) to test non-conversion code-path.

Also added new q-test, to read a parquet timestamp-file generated by an external tool, in
this case Impala.


Szehon Ho

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message