hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "genericqa (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15400) Improve S3Guard documentation on Authoritative Mode implementation
Date Tue, 07 Aug 2018 21:06:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16572339#comment-16572339
] 

genericqa commented on HADOOP-15400:
------------------------------------

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 13s{color} | {color:blue}
Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m  3s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 30s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 36m 17s{color}
| {color:green} branch has no errors when building and testing our client artifacts. {color}
|
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 30s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 19s{color}
| {color:green} patch has no errors when building and testing our client artifacts. {color}
|
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 20s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m  3s{color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-15400 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12934711/HADOOP-15400.002.patch
|
| Optional Tests |  asflicense  mvnsite  |
| uname | Linux 869c89ab9447 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 861095f |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 440 (vs. ulimit of 10000) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/14999/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Improve S3Guard documentation on Authoritative Mode implementation
> ------------------------------------------------------------------
>
>                 Key: HADOOP-15400
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15400
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.0.1
>            Reporter: Aaron Fabbri
>            Assignee: Gabor Bota
>            Priority: Minor
>         Attachments: HADOOP-15400.001.patch, HADOOP-15400.002.patch
>
>
> Part of the design of S3Guard is support for skipping the call to S3 listObjects and
serving directory listings out of the MetadataStore under certain circumstances.  This feature
is called "authoritative" mode.  I've talked to many people about this feature and it seems
to be universally confusing.
> I suggest we improve / add a section to the s3guard.md site docs elaborating on what
Authoritative Mode is.
> It is *not* treating the MetadataStore (e.g. dynamodb) as the source of truth in general.
> It *is* the ability to short-circuit S3 list objects and serve listings from the MetadataStore
in some circumstances: 
> For S3A to skip S3's list objects on some *path*, and serve it directly from the MetadataStore,
the following things must all be true:
>  # The MetadataStore implementation persists the bit {{DirListingMetadata.isAuthorititative}}
set when calling {{MetadataStore#put(DirListingMetadata)}}
>  # The S3A client is configured to allow metadatastore to be authoritative source of
a directory listing (fs.s3a.metadatastore.authoritative=true).
>  # The MetadataStore has a full listing for *path* stored in it.  This only happens
if the FS client (s3a) explicitly has stored a full directory listing with {{DirListingMetadata.isAuthorititative=true}}
before the said listing request happens.
> Note that #1 only currently happens in LocalMetadataStore. Adding support to DynamoDBMetadataStore
is covered in HADOOP-14154.
> Also, the multiple uses of the word "authoritative" are confusing. Two meanings are used:
>  1. In the FS client configuration fs.s3a.metadatastore.authoritative
>  - Behavior of S3A code (not MetadataStore)
>  - "S3A is allowed to skip S3.list() when it has full listing from MetadataStore"
> 2. MetadataStore
>  When storing a dir listing, can set a bit isAuthoritative
>  1 : "full contents of directory"
>  0 : "may not be full listing"
> Note that a MetadataStore *MAY* persist this bit. (not *MUST*).
> We should probably rename the {{DirListingMetadata.isAuthorititative}} to {{.fullListing}}
or at least put a comment where it is used to clarify its meaning.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message