hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Barna Zsombor Klara <>
Subject Re: Review Request 56334: HIVE-12767: Implement table property to address Parquet int96 timestamp bug
Date Wed, 08 Feb 2017 15:45:27 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated Feb. 8, 2017, 3:45 p.m.)

Review request for hive, Ryan Blue and Sergio Pena.


Instead of logging a warning throw an exception if an unexpected timezone value is found.

Bugs: HIVE-12767

Repository: hive-git


This is a followup on this review request:
The following exit criteria is addressed in this patch:

- Hive will read Parquet MR int96 timestamp data and adjust values using a time zone from
a table property, if set, or using the local time zone if it is absent. No adjustment will
be applied to data written by Impala.
- Hive will write Parquet int96 timestamps using a time zone adjustment from the same table
property, if set, or using the local time zone if it is absent. This keeps the data in the
table consistent.
- New tables created by Hive will set the table property to UTC if the global option to set
the property for new tables is enabled.
- Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set the property unless
the global setting to do so is enabled.
- Tables created using CREATE TABLE LIKE <OTHER TABLE> will copy the property of the
table that is copied.

To set the timezone table property, use this:
  create table tbl1 (ts timestamp) stored as parquet tblproperties (''='PST');

To set UTC as default timezone table property on new tables created, use this: 
  create table tbl2 (ts timestamp) stored as parquet;

Diffs (updated)

  common/src/java/org/apache/hadoop/hive/conf/ cb27cd6236dc1d54efd2cafda9f1cc975e9d9817

  data/files/impala_int96_timestamp.parq PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ adabe70fa8f0fe1b990c6ac578a14ff5af06fc93

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ 379a9135d9c631b2f473976b00f3dc87f9fec0c4

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ 167f9b6516ac093fa30091daf6965de25e3eccb3

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ 76d93b8e02a98c95da8a534f2820cd3e77b4bb43

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/ 604cbbcc2a9daa8594397e315cc4fd8064cc5005

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/ ac430a67682d3dcbddee89ce132fc0c1b421e368

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ PRE-CREATION

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/ 3fd75d24f3fda36967e4957e650aec19050b22f8

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/ f4621e5dbb81e8d58c4572c901ec9d1a7ca8c012

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/ 6b7b50a25e553629f0f492e964cc4913417cb500

  ql/src/test/org/apache/hadoop/hive/ql/io/parquet/ 934ae9f255d0c4ccaa422054fcc9e725873810d4

  ql/src/test/org/apache/hadoop/hive/ql/io/parquet/ 833cfdbaa2ac6a3653a8f3232344f5fb70c80686

  ql/src/test/org/apache/hadoop/hive/ql/io/parquet/convert/ PRE-CREATION

  ql/src/test/org/apache/hadoop/hive/ql/io/parquet/serde/ ec6def5b9ac5f12e6a7cb24c4f4998a6ca6b4a8e

  ql/src/test/org/apache/hadoop/hive/ql/io/parquet/timestamp/ PRE-CREATION

  ql/src/test/queries/clientpositive/parquet_int96_timestamp.q PRE-CREATION 
  ql/src/test/results/clientpositive/parquet_int96_timestamp.q.out PRE-CREATION 



qtest and unit tests added.


Barna Zsombor Klara

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message