drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vdiravka <...@git.apache.org>
Subject [GitHub] drill pull request #600: DRILL-4373: Drill and Hive have incompatible timest...
Date Fri, 28 Oct 2016 12:43:55 GMT
Github user vdiravka commented on a diff in the pull request:

    --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java
    @@ -739,30 +741,76 @@ public void runTestAndValidate(String selection, String validationSelection,
    -  Test the reading of an int96 field. Impala encodes timestamps as int96 fields
    +    Impala encodes timestamp values as int96 fields. Test the reading of an int96 field
with two converters:
    +    the first one converts parquet INT96 into drill VARBINARY and the second one (works
    +    store.parquet.reader.int96_as_timestamp option is enabled) converts parquet INT96
into drill TIMESTAMP.
       public void testImpalaParquetInt96() throws Exception {
         compareParquetReadersColumnar("field_impala_ts", "cp.`parquet/int96_impala_1.parquet`");
    +    try {
    +      test("alter session set %s = true", ExecConstants.PARQUET_READER_INT96_AS_TIMESTAMP);
    +      compareParquetReadersColumnar("field_impala_ts", "cp.`parquet/int96_impala_1.parquet`");
    +    } finally {
    +      test("alter session reset %s", ExecConstants.PARQUET_READER_INT96_AS_TIMESTAMP);
    +    }
    -  Test the reading of a binary field where data is in dicationary _and_ non-dictionary
encoded pages
    +  Test the reading of a binary field as drill varbinary where data is in dicationary
_and_ non-dictionary encoded pages
    -  public void testImpalaParquetVarBinary_DictChange() throws Exception {
    +  public void testImpalaParquetBinaryAsVarBinary_DictChange() throws Exception {
         compareParquetReadersColumnar("field_impala_ts", "cp.`parquet/int96_dict_change.parquet`");
    +  Test the reading of a binary field as drill timestamp where data is in dicationary
_and_ non-dictionary encoded pages
    +   */
    +  @Test
    +  public void testImpalaParquetBinaryAsTimeStamp_DictChange() throws Exception {
    +    final String WORKING_PATH = TestTools.getWorkingPath();
    +    final String TEST_RES_PATH = WORKING_PATH + "/src/test/resources";
    +    try {
    +      testBuilder()
    +          .sqlQuery("select int96_ts from dfs_test.`%s/parquet/int96_dict_change`", TEST_RES_PATH)
    +          .optionSettingQueriesForTestQuery(
    +              "alter session set `%s` = true", ExecConstants.PARQUET_READER_INT96_AS_TIMESTAMP)
    +          .ordered()
    +          .csvBaselineFile("testframework/testParquetReader/testInt96DictChange/q1.tsv")
    +          .baselineTypes(TypeProtos.MinorType.TIMESTAMP)
    +          .baselineColumns("int96_ts")
    +          .build().run();
    +    } finally {
    +      test("alter system reset `%s`", ExecConstants.PARQUET_READER_INT96_AS_TIMESTAMP);
    +    }
    +  }
    +  /*
          Test the conversion from int96 to impala timestamp
    -  public void testImpalaParquetTimestampAsInt96() throws Exception {
    +  public void testTimestampImpalaConvertFrom() throws Exception {
         compareParquetReadersColumnar("convert_from(field_impala_ts, 'TIMESTAMP_IMPALA')",
    +     Test reading parquet Int96 as TimeStamp and comparing obtained values with the
    +     old results (reading the same values as VarBinary and convert_fromTIMESTAMP_IMPALA
function using)
    +   */
    +  @Test
    +  public void testImpalaParquetTimestampInt96AsTimeStamp() throws Exception {
    --- End diff --
    This test compares the results between new converter (Int96 to TimeStamp) and the old
one (Int96 to VarBinary) with `convert_fromTIMESTAMP_IMPALA` function. 
    The issue was in the `ConvertFromImpalaTimestamp` [link to the code
Because the timezone changing shouldn't affect on the result timestamp values.
    I deleted timezone consideration there, so now all tests passed successfuly even across
different timezones.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message