hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13449) S3Guard: Implement DynamoDBMetadataStore.
Date Tue, 15 Nov 2016 23:32:58 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15668739#comment-15668739

Aaron Fabbri commented on HADOOP-13449:

Thanks [~liuml07], sounds good.  Can't wait to try this stuff out.

My final concern is that, the MetadataStore assumes all ancestor directories (including direct
parent directory) have been pre-created by the caller/user. I have to change the base test
MetadataStoreTestBase to make all the tests pass. We have to change LocalMetadataStore accordingly.
See my above comments.

I looked in the test code and did not see the bug you mention.  Feel free to call out a particular
line of code: I may have missed it.

This is the intention:  Say you start by doing a MetadataStore.put(PathMetadata(/a/b/file)).

That file should appear when listing /a/b.

That is, MetadataStore.listChildren(/a/b) should return {[/a/b/file], isAuthoritative=false}.
 This is used by S3AFileSystem#listStatus() to get list consistency (e.g. if /a/b/file is
not visible in s3 yet, we will add that entry).

However, there is no requirement that MetadataStore.get(/a/b) should return anything.  So
the parent directory does not need to be created, per se, but the new file *does* need to
appear when listing the parent.

If the MetadataStore contract tests do not reflect this, I am very happy to change them. 
Also you might have missed some other test code (helper functions) that explicitly create
parent dirs.  We don't want to assume creation of ancestors at all, just file to show up in
parent listing

The LocalMetadataStore ends up creating a parent dir internally, but that should be implementation
detail, and not required as above.

Does that help clarify?

> S3Guard: Implement DynamoDBMetadataStore.
> -----------------------------------------
>                 Key: HADOOP-13449
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13449
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Chris Nauroth
>            Assignee: Mingliang Liu
>         Attachments: HADOOP-13449-HADOOP-13345.000.patch, HADOOP-13449-HADOOP-13345.001.patch,
HADOOP-13449-HADOOP-13345.002.patch, HADOOP-13449-HADOOP-13345.003.patch, HADOOP-13449-HADOOP-13345.004.patch
> Provide an implementation of the metadata store backed by DynamoDB.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message