drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vdiravka <...@git.apache.org>
Subject [GitHub] drill pull request #687: DRILL-4996: Parquet Date auto-correction is not wor...
Date Mon, 19 Dec 2016 13:21:14 GMT
Github user vdiravka commented on a diff in the pull request:

    https://github.com/apache/drill/pull/687#discussion_r93028597
  
    --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestCorruptParquetDateCorrection.java
---
    @@ -201,68 +203,34 @@ public void testCorrectDatesAndExceptionWhileParsingCreatedBy()
throws Exception
             .go();
       }
     
    -  /**
    -   * Test reading a directory full of partitioned parquet files with dates, these files
have a drill version
    -   * number of 1.4.0 in their footers, so we can be certain they are corrupt. The option
to disable the
    -   * correction is passed, but it will not change the result in the case where we are
certain correction
    -   * is needed. For more info see DRILL-4203.
    -   */
    -  @Test
    -  public void testReadPartitionedOnCorruptedDates() throws Exception {
    -    try {
    -      for (String selection : new String[]{"*", "date_col"}) {
    -        // for sanity, try reading all partitions without a filter
    -        TestBuilder builder = testBuilder()
    -            .sqlQuery("select " + selection + " from table(dfs.`" + CORRUPTED_PARTITIONED_DATES_1_4_0_PATH
+ "`" +
    -                "(type => 'parquet', autoCorrectCorruptDates => false))")
    -            .unOrdered()
    -            .baselineColumns("date_col");
    -        addDateBaselineVals(builder);
    -        builder.go();
    -
    -        String query = "select " + selection + " from table(dfs.`" + CORRUPTED_PARTITIONED_DATES_1_4_0_PATH
+ "` " +
    -            "(type => 'parquet', autoCorrectCorruptDates => false))" + " where
date_col = date '1970-01-01'";
    -        // verify that pruning is actually taking place
    -        testPlanMatchingPatterns(query, new String[]{"numFiles=1"}, null);
    -
    -        // read with a filter on the partition column
    -        testBuilder()
    -            .sqlQuery(query)
    -            .unOrdered()
    -            .baselineColumns("date_col")
    -            .baselineValues(new DateTime(1970, 1, 1, 0, 0))
    -            .go();
    -      }
    -    } finally {
    -      test("alter session reset all");
    -    }
    -  }
     
       @Test
       public void testReadPartitionedOnCorruptedDates_UserDisabledCorrection() throws Exception
{
         try {
           for (String selection : new String[]{"*", "date_col"}) {
    -        // for sanity, try reading all partitions without a filter
    -        TestBuilder builder = testBuilder()
    -            .sqlQuery("select " + selection + " from table(dfs.`" + CORRUPTED_PARTITIONED_DATES_1_2_PATH
+ "`" +
    -                "(type => 'parquet', autoCorrectCorruptDates => false))")
    -            .unOrdered()
    -            .baselineColumns("date_col");
    -        addCorruptedDateBaselineVals(builder);
    -        builder.go();
    -
    -        String query = "select " + selection + " from table(dfs.`" + CORRUPTED_PARTITIONED_DATES_1_2_PATH
+ "` " +
    -            "(type => 'parquet', autoCorrectCorruptDates => false))" + " where
date_col = cast('15334-03-17' as date)";
    -        // verify that pruning is actually taking place
    -        testPlanMatchingPatterns(query, new String[]{"numFiles=1"}, null);
    -
    -        // read with a filter on the partition column
    -        testBuilder()
    -            .sqlQuery(query)
    -            .unOrdered()
    -            .baselineColumns("date_col")
    -            .baselineValues(new DateTime(15334, 03, 17, 0, 0))
    -            .go();
    +        for (String table : new String[]{CORRUPTED_PARTITIONED_DATES_1_2_PATH, CORRUPTED_PARTITIONED_DATES_1_4_0_PATH})
{
    --- End diff --
    
    It is necessary to keep indent (211 line) after `try` block and then (212 line) after
`for` block.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message