hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-21291) Restore historical way of handling timestamps in Avro while keeping the new semantics at the same time
Date Tue, 23 Apr 2019 09:21:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823879#comment-16823879
] 

Hive QA commented on HIVE-21291:
--------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 48s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m  9s{color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 59s{color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  3m 17s{color}
| {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 32s{color} | {color:blue}
common in master has 63 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 41s{color} | {color:blue}
serde in master has 197 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  2s{color} | {color:blue}
ql in master has 2256 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  8m 13s{color} |
{color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 25s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 48s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 59s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 59s{color} | {color:green}
the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 16s{color} | {color:red}
serde: The patch generated 2 new + 131 unchanged - 0 fixed = 133 total (was 131) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  2m  6s{color} | {color:red}
root: The patch generated 2 new + 588 unchanged - 0 fixed = 590 total (was 588) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 46s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  7m 57s{color} |
{color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 13s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 10s{color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19)
x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17010/dev-support/hive-personality.sh
|
| git revision | master / b58d50c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-17010/yetus/diff-checkstyle-serde.txt
|
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-17010/yetus/diff-checkstyle-root.txt
|
| modules | C: common serde ql . U: . |
| Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17010/yetus.txt |
| Powered by | Apache Yetus    http://yetus.apache.org |


This message was automatically generated.



> Restore historical way of handling timestamps in Avro while keeping the new semantics
at the same time
> ------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-21291
>                 URL: https://issues.apache.org/jira/browse/HIVE-21291
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Zoltan Ivanfi
>            Assignee: Karen Coppage
>            Priority: Major
>         Attachments: HIVE-21291.1.patch, HIVE-21291.2.patch, HIVE-21291.3.patch, HIVE-21291.4.patch,
HIVE-21291.4.patch, HIVE-21291.5.patch, HIVE-21291.6.patch
>
>
> This sub-task is for implementing the Avro-specific parts of the following plan:
> h1. Problem
> Historically, the semantics of the TIMESTAMP type in Hive depended on the file format.
Timestamps in Avro, Parquet and RCFiles with a binary SerDe had _Instant_ semantics, while
timestamps in ORC, textfiles and RCFiles with a text SerDe had _LocalDateTime_ semantics.
> The Hive community wanted to get rid of this inconsistency and have _LocalDateTime_ semantics
in Avro, Parquet and RCFiles with a binary SerDe as well. *Hive 3.1 turned off normalization
to UTC* to achieve this. While this leads to the desired new semantics, it also leads to incorrect
results when new Hive versions read timestamps written by old Hive versions or when old Hive
versions or any other component not aware of this change (including legacy Impala and Spark
versions) read timestamps written by new Hive versions.
> h1. Solution
> To work around this issue, Hive *should restore the practice of normalizing to UTC* when
writing timestamps to Avro, Parquet and RCFiles with a binary SerDe. In itself, this would
restore the historical _Instant_ semantics, which is undesirable. In order to achieve the
desired _LocalDateTime_ semantics in spite of normalizing to UTC, newer Hive versions should
record the session-local local time zone in the file metadata fields serving arbitrary key-value
storage purposes.
> When reading back files with this time zone metadata, newer Hive versions (or any other
new component aware of this extra metadata) can achieve _LocalDateTime_ semantics by *converting
from UTC to the saved time zone (instead of to the local time zone)*. Legacy components that
are unaware of the new metadata can read the files without any problem and the timestamps
will show the historical Instant behaviour to them.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message