drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-3867) Store relative paths in metadata file
Date Fri, 05 May 2017 23:51:04 GMT

    [ https://issues.apache.org/jira/browse/DRILL-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999146#comment-15999146
] 

ASF GitHub Bot commented on DRILL-3867:
---------------------------------------

Github user paul-rogers commented on a diff in the pull request:

    https://github.com/apache/drill/pull/824#discussion_r115105413
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/Metadata.java
---
    @@ -526,6 +534,48 @@ private void writeFile(ParquetTableMetadataDirs parquetTableMetadataDirs,
Path p
       }
     
       /**
    +   * Serializer for ParquetPath. Writes the path relative to the root path
    +   */
    +  private static class ParquetPathSerializer extends StdSerializer<ParquetPath>
{
    +    private final String rootPath;
    +
    +    ParquetPathSerializer(String rootPath) {
    +      super(ParquetPath.class);
    +      this.rootPath = rootPath;
    +    }
    +
    +    @Override
    +    public void serialize(ParquetPath parquetPath, JsonGenerator jsonGenerator, SerializerProvider
serializerProvider) throws IOException, JsonGenerationException {
    +      Preconditions.checkState(parquetPath.getFullPath().startsWith(rootPath), String.format("Path
%s is not a subpath of %s", parquetPath.getFullPath(), rootPath));
    +      String relativePath = parquetPath.getFullPath().replaceFirst(rootPath, "");
    --- End diff --
    
    Java defines a Path abstraction that will compute relative paths.
    
    An [old fashioned way](http://stackoverflow.com/questions/204784/how-to-construct-a-relative-path-in-java-from-two-absolute-paths-or-urls):
    ```
    String relative = new File(base).toURI().relativize(new File(path).toURI()).getPath();
    ```
    
    From the same post, the Java 7 way:
    ```
            Path pathAbsolute = Paths.get("/var/data/stuff/xyz.dat");
            Path pathBase = Paths.get("/var/data");
            Path pathRelative = pathBase.relativize(pathAbsolute);
    ```
    
    Hadoop's path may have something similar.


> Store relative paths in metadata file
> -------------------------------------
>
>                 Key: DRILL-3867
>                 URL: https://issues.apache.org/jira/browse/DRILL-3867
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Metadata
>    Affects Versions: 1.2.0
>            Reporter: Rahul Challapalli
>            Assignee: Vitalii Diravka
>             Fix For: Future
>
>
> git.commit.id.abbrev=cf4f745
> git.commit.time=29.09.2015 @ 23\:19\:52 UTC
> The below sequence of steps reproduces the issue
> 1. Create the cache file
> {code}
> 0: jdbc:drill:zk=10.10.103.60:5181> refresh table metadata dfs.`/drill/testdata/metadata_caching/lineitem`;
> +-------+-------------------------------------------------------------------------------------+
> |  ok   |                                       summary                             
         |
> +-------+-------------------------------------------------------------------------------------+
> | true  | Successfully updated metadata for table /drill/testdata/metadata_caching/lineitem.
 |
> +-------+-------------------------------------------------------------------------------------+
> 1 row selected (1.558 seconds)
> {code}
> 2. Move the directory
> {code}
> hadoop fs -mv /drill/testdata/metadata_caching/lineitem /drill/
> {code}
> 3. Now run a query on top of it
> {code}
> 0: jdbc:drill:zk=10.10.103.60:5181> select * from dfs.`/drill/lineitem` limit 1;
> Error: SYSTEM ERROR: FileNotFoundException: Requested file maprfs:///drill/testdata/metadata_caching/lineitem/2006/1
does not exist.
> [Error Id: b456d912-57a0-4690-a44b-140d4964903e on pssc-66.qa.lab:31010] (state=,code=0)
> {code}
> This is obvious given the fact that we are storing absolute file paths in the cache file



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message