hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
Date Tue, 31 Jul 2018 05:47:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16562984#comment-16562984

Aaron Fabbri edited comment on HADOOP-14154 at 7/31/18 5:46 AM:

Thanks for the patch

edit: I completely read the wrong patch.. ignore previous comment. Will re-review in the morning.

was (Author: fabbri):
Thanks for the patch.

 * Added {{PathMetadata#isAuthoritativeDir}} because {{DynamoDBMetadataStore}} stores {{PathMetadata}} in
ddb, so that was a logical choice.

I did not see this in your diff, it may be missing some files?

This may be a quick way to get the code working because PathMetadata are translated DDB rows,
but I think it confuses the MetadataStore API.  I think the API should probably remain as
is (you set Authoritative bit on directories, not files). This probably means you need to
change the PathMetadataDynamoDBTranslation logic a bit, though.

Also the testAuthoritative case you added; should that go in the base class instead of the
TestDynamoDBMetadataStore subclass?  I think it should.  Any MetadataStore should behave the
same if it supports persisting the auth bit (they all will now, except for Null of course).
There is already some code there, though.  See testListChildrenAuthoritative()

--- hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java
+++ hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java
@@ -188,6 +188,7 @@
   private String tableName;
   private Configuration conf;
   private String username;
+  private boolean allowAuthoritative;

The MetadataStore just persists the auth bit on directories.  The FS (S3A) is what decides
which listing is a full/complete/authoritative listing.  So, I don't think you need this here.
@@ -365,7 +374,7 @@ public DirListingMetadata listChildren(Path path) throws IOException {
       return (metas.isEmpty() && get(path) == null)
           ? null
-          : new DirListingMetadata(path, metas, false);
+          : new DirListingMetadata(path, metas, allowAuthoritative);
     } catch (AmazonClientException e) {
       throw translateException("listChildren", path, e);
A DirListingMetadata is authoritative=true iff (if and only if) the FS set it when it previously
called put().  Returning true based on the config is not correct here.

That config option "allow authoritative" should only be visible in the FS (S3A) code; it declares
that S3A *may* treat MS results as authoritative if the bit is set.  The MetadataStore's job
is simply to return (persist) what it was told earlier by the FS when it returns a DirListingMetadata.

So if MS.put(/some/dir, auth=true)
then MS.get(/some/dir) can return with auth=true (assuming a prune() or something else has
not invalidated the listing for that dir)

but if MS.put(/some/dir, auth=false)
then MS.get(/some/dir) must return with auth=false.

This behavior is completely independent from the value of fs.s3a.metadatastore.authoritative
(that is only used by the FS to skip round trips to S3 for listings)

Thanks for digging into this feature--I know it is confusing but I hope this helps some.

> Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
> -----------------------------------------------------------------------------
>                 Key: HADOOP-14154
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14154
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.0.0-beta1
>            Reporter: Rajesh Balamohan
>            Assignee: Gabor Bota
>            Priority: Minor
>         Attachments: HADOOP-14154-HADOOP-13345.001.patch, HADOOP-14154-HADOOP-13345.002.patch,
HADOOP-14154-spec-001.pdf, HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch
> Add support for "authoritative mode" for DynamoDBMetadataStore.
> The missing feature is to persist the bit set in {{DirListingMetadata.isAuthoritative}}. 
> This topic has been super confusing for folks so I will also file a documentation Jira
to explain the design better.
> We may want to also rename the DirListingMetadata.isAuthoritative field to .isFullListing
to eliminate the multiple uses and meanings of the word "authoritative".

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message