hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
Date Thu, 19 Jul 2018 22:37:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16549960#comment-16549960

Aaron Fabbri commented on HADOOP-14154:

Doc looks great [~gabor.bota]. One minor comment:

Authoritative directory listings (isAuthoritative bit)
 * Tells if the stored directory listing metadata is complete.
 * This is set by the internal logic of the metadata store in the DirListingMetadata(org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata).
The last point, I suggest clarifying to:
This is set by the FileSystem client (e.g. s3a) via the DirListingMetadata class
Basically, the MetadataStore only knows what the FS client tells it.

We might want to also talk about adding a "entry last written timestamp" column as well while
we are at it.  I created HADOOP-15621 to describe the need for this.  Might be easier to add
this to your doc here and we can review all the compatibility concerns together in one place?

> Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
> -----------------------------------------------------------------------------
>                 Key: HADOOP-14154
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14154
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.0.0-beta1
>            Reporter: Rajesh Balamohan
>            Assignee: Gabor Bota
>            Priority: Minor
>         Attachments: HADOOP-14154-HADOOP-13345.001.patch, HADOOP-14154-HADOOP-13345.002.patch,
> Add support for "authoritative mode" for DynamoDBMetadataStore.
> The missing feature is to persist the bit set in {{DirListingMetadata.isAuthoritative}}. 
> This topic has been super confusing for folks so I will also file a documentation Jira
to explain the design better.
> We may want to also rename the DirListingMetadata.isAuthoritative field to .isFullListing
to eliminate the multiple uses and meanings of the word "authoritative".

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message