hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14154) Set isAuthoritative flag when creating DirListingMetadata in DynamoDBMetaStore
Date Thu, 09 Mar 2017 22:53:38 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15904022#comment-15904022

Aaron Fabbri commented on HADOOP-14154:

Would that isAuthoritative flag have to be setup by higher level applications like Pig/Hive/MR?

No, it is internal to S3A.  S3A can tell when it has the full listing for a directory, and
simply conveys that to the MetadataStore by setting the isAuthoritative bit.

e.g. in {{S3Guard#dirListingUnion(..)}}, it always sets the flag when it puts the listing
into the MetadataStore, since this function always has the full listing for the directory:

  dirMeta.setAuthoritative(true); // This is the full directory contents

> Set isAuthoritative flag when creating DirListingMetadata in DynamoDBMetaStore
> ------------------------------------------------------------------------------
>                 Key: HADOOP-14154
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14154
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Rajesh Balamohan
>            Priority: Minor
>         Attachments: HADOOP-14154-HADOOP-13345.001.patch, HADOOP-14154-HADOOP-13345.002.patch
> Currently {{DynamoDBMetaStore::listChildren}} does not populate {{isAuthoritative}} flag
when creating {{DirListingMetadata}}. 
> This causes additional S3 lookups even when users have enabled {{fs.s3a.metadatastore.authoritative}}.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message