hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabor Bota (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
Date Tue, 14 Aug 2018 14:58:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579915#comment-16579915

Gabor Bota commented on HADOOP-14154:

I've debugged this, and the issue is that we never set the {{dirMeta.setAuthoritative(true);}}
anywhere else, and the implementation populates every directory with every entry before {{S3Guard#dirListingUnion}}
runs. This causes {{changed}} flag inside {{dirListingUnion}} to be false all the time during
the test - the listing will be the same and the {{dirMeta}} will stay isAuthoritative=false.

What do you think about this [~fabbri]? Do we need to change that?

> Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
> -----------------------------------------------------------------------------
>                 Key: HADOOP-14154
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14154
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.0.0-beta1
>            Reporter: Rajesh Balamohan
>            Assignee: Gabor Bota
>            Priority: Minor
>         Attachments: HADOOP-14154-HADOOP-13345.001.patch, HADOOP-14154-HADOOP-13345.002.patch,
HADOOP-14154-spec-001.pdf, HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch,
HADOOP-14154.003.patch, HADOOP-14154.004.patch, all-logs-v2.txt, perf-eval-v1.diff, perf-eval-v2.diff,
run-dir-perf-itest-v2.sh, run-dir-perf-itest.sh
> Add support for "authoritative mode" for DynamoDBMetadataStore.
> The missing feature is to persist the bit set in {{DirListingMetadata.isAuthoritative}}. 
> This topic has been super confusing for folks so I will also file a documentation Jira
to explain the design better.
> We may want to also rename the DirListingMetadata.isAuthoritative field to .isFullListing
to eliminate the multiple uses and meanings of the word "authoritative".

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message