drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vdiravka <...@git.apache.org>
Subject [GitHub] drill pull request #877: DRILL-5660: Drill 1.10 queries fail due to Parquet ...
Date Wed, 26 Jul 2017 08:57:32 GMT
Github user vdiravka commented on a diff in the pull request:

    https://github.com/apache/drill/pull/877#discussion_r129486097
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/Metadata.java
---
    @@ -1851,9 +1922,73 @@ private static String relativize(String baseDir, String childPath)
{
               .relativize(fullPathWithoutSchemeAndAuthority.toUri()));
           if (relativeFilePath.isAbsolute()) {
             throw new IllegalStateException(String.format("Path %s is not a subpath of %s.",
    -            basePathWithoutSchemeAndAuthority.toUri().toString(), fullPathWithoutSchemeAndAuthority.toUri().toString()));
    +            basePathWithoutSchemeAndAuthority.toUri().getPath(), fullPathWithoutSchemeAndAuthority.toUri().getPath()));
    +      }
    +      return relativeFilePath.toUri().getPath();
    +    }
    +  }
    +
    +  /**
    +   * Supported metadata versions.
    +   * <p>
    +   * Note: keep them synchronized with {@link ParquetTableMetadataBase} versions
    +   */
    +  public static class MetadataVersion {
    +
    +    /**
    +     * Version 1: Introduces parquet file metadata caching.<br>
    +     * See DRILL-2743
    +     */
    +    public static final String V1 = "v1";
    +    /**
    +     * Version 2: Metadata cache file size is reduced.<br>
    +     * See DRILL-4053
    +     */
    +    public static final String V2 = "v2";
    +    /**
    +     * Version 3: Difference between v3 and v2 : min/max, type_length, precision, scale,
repetitionLevel, definitionLevel.<br>
    +     * Filter pushdown for Parquet is implemented. <br>
    +     * See DRILL-1950
    +     */
    +    public static final String V3 = "v3";
    +    /**
    +     * Version 3.1: Absolute paths of files and directories are replaced with relative
ones.<br>
    +     * See DRILL-3867
    +     */
    +    public static final String V3_1 = "v3.1";
    +
    +
    +    /**
    +     * All historical versions of the Drill metadata cache files
    +     */
    +    public static final List<String> SUPPORTED_VERSIONS = Lists.newArrayList(V1,
V2, V3, V3_1);
    +
    +    /**
    +     * @param metadataVersion parquet metadata version
    +     * @return true if metadata version is supported, false otherwise
    +     */
    +    public static boolean isVersionSupported(String metadataVersion) {
    +      return SUPPORTED_VERSIONS.contains(metadataVersion);
    +    }
    +
    +    /**
    +     * Helper compare method similar to {@link java.util.Comparator#compare}
    +     *
    +     * @param metadataVersion1 the first metadata version to be compared
    +     * @param metadataVersion2 the second metadata version to be compared
    +     * @return a negative integer, zero, or a positive integer as the
    +     *         first argument is less than, equal to, or greater than the
    +     *         second.
    +     */
    +    public static int compare(String metadataVersion1, String metadataVersion2) {
    --- End diff --
    
    Added new `VersionParser` class, which allow to parse parquet metadata version string.
Added test cases for different correct and wrong parquet metadata versions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message