hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergio Pena" <sergio.p...@cloudera.com>
Subject Review Request 41821: Implement table property to address Parquet int96 timestamp bug
Date Thu, 31 Dec 2015 00:22:47 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41821/
-----------------------------------------------------------

Review request for hive, Ryan Blue, Mohammad Islam, Reuben Kuhnert, and Szehon Ho.


Bugs: HIVE-12767
    https://issues.apache.org/jira/browse/HIVE-12767


Repository: hive-git


Description
-------

The following exit criteria is addressed in this patch:

* Hive will read Parquet MR int96 timestamp data and adjust values using a time zone from
a table property, if set, or using the local time zone if it is absent. No adjustment will
be applied to data written by Impala.

* Hive will write Parquet int96 timestamps using a time zone adjustment from the same table
property, if set, or using the local time zone if it is absent. This keeps the data in the
table consistent.

* New tables created by Hive will set the table property to UTC if the global option to set
the property for new tables is enabled.
  * Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set the property
unless the global setting to do so is enabled.
  * Tables created using CREATE TABLE LIKE <OTHER TABLE> will copy the property of the
table that is copied.


Diffs
-----

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 31f063406a85d82189a3ff4f2d66c001be9350f1

  data/files/impala_int96_timestamp.parq PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 4fb6c00ba16b5514463fa18c8655f7c9811fec21

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetOutputFormat.java bfb48a987ce89a373f3da63c9162546c6eda43a9

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ETypeConverter.java ec0dd818f688ab92feb46be4fb6040ede5ac756a

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/DataWritableReadSupport.java 53f3b72b790d87a75a7cd1d77d8f011c29c41188

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/ParquetRecordReaderWrapper.java 74a1a82047613189678716f765bfaa9ac39b7618

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetTableUtils.java PRE-CREATION

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/NanoTimeUtils.java aace48ee7d145d199163286d21e4ee7694140d6f

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriteSupport.java f4621e5dbb81e8d58c4572c901ec9d1a7ca8c012

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriter.java 69272dc41dbc5fe29ab4c98e730b591c28f3a297

  ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestDataWritableWriter.java 70491390ba2b90f32ef9963be7b19e57672241f3

  ql/src/test/org/apache/hadoop/hive/ql/io/parquet/convert/TestETypeConverter.java PRE-CREATION

  ql/src/test/org/apache/hadoop/hive/ql/io/parquet/serde/TestParquetTimestampUtils.java ec6def5b9ac5f12e6a7cb24c4f4998a6ca6b4a8e

  ql/src/test/org/apache/hadoop/hive/ql/io/parquet/timestamp/TestParquetTimestampConverter.java
PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_int96_timestamp.q PRE-CREATION 
  ql/src/test/results/clientpositive/parquet_int96_timestamp.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/41821/diff/


Testing
-------

Added unit and q-tests:
  ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestDataWritableWriter.java
  ql/src/test/org/apache/hadoop/hive/ql/io/parquet/convert/TestETypeConverter.java
  ql/src/test/org/apache/hadoop/hive/ql/io/parquet/serde/TestParquetTimestampUtils.java
  ql/src/test/org/apache/hadoop/hive/ql/io/parquet/timestamp/TestParquetTimestampConverter.java
  ql/src/test/queries/clientpositive/parquet_int96_timestamp.q


Thanks,

Sergio Pena


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message