drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] (DRILL-5034) Select timestamp from hive generated parquet always return in UTC
Date Sun, 29 Jan 2017 17:56:44 GMT
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<html xmlns="http://www.w3.org/1999/xhtml"> 
    <head> 
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> 
        <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0"
/> <base href="https://issues.apache.org/jira" /> 
        <title>Message Title</title> 
    </head> 
    <body class="jira" style="color: #333; font-family: Arial, sans-serif; font-size: 14px;
line-height: 1.429"> 
        <table id="background-table" cellpadding="0" cellspacing="0" width="100%" style="border-collapse:
collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; background-color: #f5f5f5; border-collapse:
collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt"> 
            <!-- header here --> 
            <tr> 
                <td id="header-pattern-container" style="padding: 0px; border-collapse:
collapse; padding: 10px 20px"> 
                    <table id="header-pattern" cellspacing="0" cellpadding="0" border="0"
style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt"> 
                        <tr> 
                            <td id="header-avatar-image-container" valign="top" style="padding:
0px; border-collapse: collapse; vertical-align: top; width: 32px; padding-right: 8px">
<img id="header-avatar-image" class="image_fix" src="cid:jira-generated-image-avatar-githubbot-be1080d6-a529-40ec-9060-584728d30838"
height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top" /> 
                            </td> 
                            <td id="header-text-container" valign="middle" style="padding:
0px; border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; font-size:
14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px"> <a class="user-hover"
rel="githubbot" id="email_githubbot" href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=githubbot"
style="color:#3b73af;; color: #3b73af; text-decoration: none">ASF GitHub Bot</a>
<strong>commented</strong> on <a href="https://issues.apache.org/jira/browse/DRILL-5034"
style="color: #3b73af; text-decoration: none"><img src="cid:jira-generated-image-static-bug-e3286b7c-0c8e-4fc8-a951-bc198eafb9f0"
height="16" width="16" border="0" align="absmiddle" alt="Bug" /> DRILL-5034</a> 
                            </td> 
                        </tr> 
                    </table> 
                </td> 
            </tr> 
            <tr> 
                <td id="email-content-container" style="padding: 0px; border-collapse:
collapse; padding: 0 20px"> 
                    <table id="email-content-table" cellspacing="0" cellpadding="0" border="0"
width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt;
border-spacing: 0; border-collapse: separate"> 
                        <tr> 
                            <!-- there needs to be content in the cell for it to render
in some clients --> 
                            <td class="email-content-rounded-top mobile-expand" style="padding:
0px; border-collapse: collapse; color: #fff; padding: 0 15px 0 16px; height: 15px; background-color:
#fff; border-left: 1px solid #ccc; border-top: 1px solid #ccc; border-right: 1px solid #ccc;
border-bottom: 0; border-top-right-radius: 5px; border-top-left-radius: 5px; height: 10px;
line-height: 10px; padding: 0 15px 0 16px; mso-line-height-rule: exactly">
                                &nbsp;
                            </td> 
                        </tr> 
                        <tr> 
                            <td class="email-content-main mobile-expand " style="padding:
0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc;
border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff"> 
                                <table class="page-title-pattern" cellspacing="0" cellpadding="0"
border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace:
0pt"> 
                                    <tr> 
                                        <td style="vertical-align: top;; padding: 0px;
border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: 30px; mso-line-height-rule:
exactly" class="page-title-pattern-header-container"> <span class="page-title-pattern-header"
style="font-family: Arial, sans-serif; padding: 0; font-size: 20px; line-height: 30px; mso-text-raise:
2px; mso-line-height-rule: exactly; vertical-align: middle"> <a href="https://issues.apache.org/jira/browse/DRILL-5034"
style="color: #3b73af; text-decoration: none">Re: Select timestamp from hive generated
parquet always return in UTC</a> </span> 
                                        </td> 
                                    </tr> 
                                </table> 
                            </td> 
                        </tr> 
                        <tr> 
                            <td id="text-paragraph-pattern-top" class="email-content-main
mobile-expand  comment-top-pattern" style="padding: 0px; border-collapse: collapse; border-left:
1px solid #ccc; border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0
15px 0 16px; background-color: #fff; border-bottom: none; padding-bottom: 0"> 
                                <table class="text-paragraph-pattern" cellspacing="0" cellpadding="0"
border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace:
0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule:
exactly; mso-text-raise: 2px"> 
                                    <tr> 
                                        <td class="text-paragraph-pattern-container mobile-resize-text
" style="padding: 0px; border-collapse: collapse; padding: 0 0 10px 0"> 
                                            <p style="margin: 10px 0 0 0">Github user
vdiravka commented on a diff in the pull request:</p> 
                                            <p style="margin: 10px 0 0 0"> <a href="https://github.com/apache/drill/pull/656#discussion_r98358344"
class="external-link" rel="nofollow" style="color: #3b73af; text-decoration: none">https://github.com/apache/drill/pull/656#discussion_r98358344</a></p>

                                            <p style="margin: 10px 0 0 0"> — Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetReaderUtility.java
—<br /> @@ -323,18 +323,28 @@ public static DateCorruptionStatus checkForCorruptDateValuesInStatistics(Parquet</p>

                                            <ul> 
                                                <li>
                                                    @param binaryTimeStampValue
                                                </li> 
                                                <li>
                                                    hive, impala timestamp values with nanoseconds
precision
                                                </li> 
                                                <li>
                                                    are stored in parquet Binary as INT96
(12 constant bytes)
                                                </li> 
                                            </ul> 
                                            <ul class="alternate" type="square"> 
                                                <li>
                                                    *<br /> + * @param retainLocalTimezone<br
/> + * parquet files don't keep local timeZone according to the<br /> + * &lt;a
href=&quot;https://github.com/Parquet/parquet-format/blob/master/LogicalTypes.md#timestamp&quot;&gt;Parquet
spec&lt;/a&gt;,<br /> + * but some tools (hive, for example) retain local timezone
for parquet files by default<br /> + * Note: Impala doesn't retain local timezone by
default
                                                </li> 
                                            </ul> 
                                            <ul> 
                                                <li>
                                                    @return Unix Timestamp - the number of
milliseconds since January 1, 1970, 00:00:00 GMT
                                                </li> 
                                                <li>
                                                    represented by @param binaryTimeStampValue
.<br /> */
                                                </li> 
                                            </ul> 
                                            <ul class="alternate" type="square"> 
                                                <li>
                                                    public static long getDateTimeValueFromBinary(Binary
binaryTimeStampValue) {<br /> + public static long getDateTimeValueFromBinary(Binary
binaryTimeStampValue, boolean retainLocalTimezone) {<br /> // This method represents
binaryTimeStampValue as ByteBuffer, where timestamp is stored as sum of<br /> // julian
day number (32-bit) and nanos of day (64-bit)<br /> NanoTime nt = NanoTime.fromBinary(binaryTimeStampValue);<br
/> int julianDay = nt.getJulianDay();<br /> long nanosOfDay = nt.getTimeOfDayNanos();
                                                </li> 
                                                <li>
                                                    return (julianDay - JULIAN_DAY_NUMBER_FOR_UNIX_EPOCH)
* DateTimeConstants.MILLIS_PER_DAY<br /> + long dateTime = (julianDay - JULIAN_DAY_NUMBER_FOR_UNIX_EPOCH)
* DateTimeConstants.MILLIS_PER_DAY<br /> + nanosOfDay / NANOS_PER_MILLISECOND;<br
/> + if (retainLocalTimezone) {<br /> + return new org.joda.time.DateTime(dateTime,
org.joda.time.chrono.JulianChronology.getInstance())<br /> + .withZoneRetainFields(org.joda.time.DateTimeZone.UTC).getMillis();

                                                    <ul class="alternate" type="square">

                                                        <li> 
                                                            <ul class="alternate" type="square">

                                                                <li>
                                                                    End diff –
                                                                </li> 
                                                            </ul> 
                                                        </li> 
                                                    </ul> 
                                                </li> 
                                            </ul> 
                                            <p style="margin: 10px 0 0 0"> `withZoneRetainFields`
method calculates the difference between local timezone and UTC (parameter of that method)
and returns original dateTime with a shift of that difference. This approach is used frequently
in drill code.<br /> But thinking a little more on this I decided that it is possible
to use more simpler statement, without creating DateTime object. <br /> `DateTimeZone.getDefault().convertUTCToLocal(dateTime)`.
I think it's more clear.</p> 
                                        </td> 
                                    </tr> 
                                </table> 
                            </td> 
                        </tr> 
                        <tr> 
                            <td class="email-content-main mobile-expand " style="padding:
0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc;
border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff"> 
                                <table id="actions-pattern" cellspacing="0" cellpadding="0"
border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace:
0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule:
exactly; mso-text-raise: 1px"> 
                                    <tr> 
                                        <td id="actions-pattern-container" valign="middle"
style="padding: 0px; border-collapse: collapse; padding: 10px 0 10px 24px; vertical-align:
middle; padding-left: 0"> 
                                            <table align="left" style="border-collapse:
collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt"> 
                                                <tr> 
                                                    <td class="actions-pattern-action-icon-container"
style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size:
14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0px; vertical-align:
middle"> <a href="https://issues.apache.org/jira/browse/DRILL-5034#add-comment" target="_blank"
title="Add Comment" style="color: #3b73af; text-decoration: none"> <img class="actions-pattern-action-icon-image"
src="cid:jira-generated-image-static-comment-icon-15151520-630b-4755-9b80-a44f110d8e92" alt="Add
Comment" title="Add Comment" height="16" width="16" border="0" style="vertical-align: middle"
/> </a> 
                                                    </td> 
                                                    <td class="actions-pattern-action-text-container"
style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size:
14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left:
5px"> <a href="https://issues.apache.org/jira/browse/DRILL-5034#add-comment" target="_blank"
title="Add Comment" style="color: #3b73af; text-decoration: none">Add Comment</a>

                                                    </td> 
                                                </tr> 
                                            </table> 
                                        </td> 
                                    </tr> 
                                </table> 
                            </td> 
                        </tr> 
                        <!-- there needs to be content in the cell for it to render in
some clients --> 
                        <tr> 
                            <td class="email-content-rounded-bottom mobile-expand" style="padding:
0px; border-collapse: collapse; color: #fff; padding: 0 15px 0 16px; height: 5px; line-height:
5px; background-color: #fff; border-top: 0; border-left: 1px solid #ccc; border-bottom: 1px
solid #ccc; border-right: 1px solid #ccc; border-bottom-right-radius: 5px; border-bottom-left-radius:
5px; mso-line-height-rule: exactly">
                                &nbsp;
                            </td> 
                        </tr> 
                    </table> 
                </td> 
            </tr> 
            <tr> 
                <td id="footer-pattern" style="padding: 0px; border-collapse: collapse;
padding: 12px 20px"> 
                    <table id="footer-pattern-container" cellspacing="0" cellpadding="0"
border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">

                        <tr> 
                            <td id="footer-pattern-text" class="mobile-resize-text" width="100%"
style="padding: 0px; border-collapse: collapse; color: #999; font-size: 12px; line-height:
18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px">
                                 This message was sent by Atlassian JIRA <span id="footer-build-information">(v6.3.15#6346-<span
title="dbc023dd75cecacf443c4b235f66124b15f5c5fe" data-commit-id="dbc023dd75cecacf443c4b235f66124b15f5c5fe}">sha1:dbc023d</span>)</span>

                            </td> 
                            <td id="footer-pattern-logo-desktop-container" valign="top"
style="padding: 0px; border-collapse: collapse; padding-left: 20px; vertical-align: top">

                                <table style="border-collapse: collapse; mso-table-lspace:
0pt; mso-table-rspace: 0pt"> 
                                    <tr> 
                                        <td id="footer-pattern-logo-desktop-padding" style="padding:
0px; border-collapse: collapse; padding-top: 3px"> <img id="footer-pattern-logo-desktop"
src="cid:jira-generated-image-static-footer-desktop-logo-85c83c2f-c93e-45cd-9608-7d19e232a465"
alt="Atlassian logo" title="Atlassian logo" width="169" height="36" class="image_fix" />

                                        </td> 
                                    </tr> 
                                </table> 
                            </td> 
                        </tr> 
                    </table> 
                </td> 
            </tr> 
        </table>   
    </body>
</html>
Mime
View raw message