drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-4980) Upgrading of the approach of parquet date correctness status detection
Date Wed, 09 Nov 2016 16:58:59 GMT

    [ https://issues.apache.org/jira/browse/DRILL-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15651439#comment-15651439
] 

ASF GitHub Bot commented on DRILL-4980:
---------------------------------------

Github user paul-rogers commented on a diff in the pull request:

    https://github.com/apache/drill/pull/644#discussion_r87232692
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetReaderUtility.java
---
    @@ -189,10 +194,14 @@ public static DateCorruptionStatus detectCorruptDates(ParquetMetadata
footer,
     
         String createdBy = footer.getFileMetaData().getCreatedBy();
         String drillVersion = footer.getFileMetaData().getKeyValueMetaData().get(ParquetRecordWriter.DRILL_VERSION_PROPERTY);
    -    String isDateCorrect = footer.getFileMetaData().getKeyValueMetaData().get(ParquetRecordWriter.IS_DATE_CORRECT_PROPERTY);
    +    String writerVersion = footer.getFileMetaData().getKeyValueMetaData().get(ParquetRecordWriter.WRITER_VERSION_PROPERTY);
    +    // This flag can be present in parquet files which were generated with 1.9.0-SNAPSHOT
drill version
    +    final String isDateCorrectFlag = "is.date.correct";
    --- End diff --
    
    Maybe here you want to special case the "is.date.correct" flag.
    
    1) If the writer version is present, use it.
    2) If "is.date.correct" is present, set the writer version to "2".
    3) If neither are present, set the writer version to 1.
    
    That way we don't have to have (much) extra logic for the "is.date.correct" handling.


> Upgrading of the approach of parquet date correctness status detection
> ----------------------------------------------------------------------
>
>                 Key: DRILL-4980
>                 URL: https://issues.apache.org/jira/browse/DRILL-4980
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Storage - Parquet
>    Affects Versions: 1.8.0
>            Reporter: Vitalii Diravka
>            Assignee: Vitalii Diravka
>             Fix For: 1.9.0
>
>
> This jira is an addition for the [DRILL-4203|https://issues.apache.org/jira/browse/DRILL-4203].
> The date correctness label for the new generated parquet files should be upgraded. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message