hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15525) s3a: clarify / improve support for mixed ACL buckets
Date Sat, 09 Jun 2018 13:25:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506982#comment-16506982

Steve Loughran commented on HADOOP-15525:

As you note, *most of this is done* in HADOOP-15176. See [Assumed Roles|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/assumed_roles.md];
while {{org.apache.hadoop.fs.s3a.auth.RoleModel}} gives you a declarative model to set up
a role for testing; makes it relatively straightforward to generate S3A filesystem instances
with reduced rights and see that things fall back.

# the post-create actions are happy with perm failures, they just end up leaving markers around.
So I'm not worried about 
# the pre-create actions look for the parent entry & make sure it isn't a file; maybe
handle a permission failure there? 
# big issue is rename, especially with S3Guard involved

HADOOP-15183 covers that. If you try to rename a directory and you don't have read or delete
perms to a path underneath, rename() fails *without updating s3guard as to its current state*.
It needs to sync up & add tombstones for all deleted entries, new entries for created

* Delete will also need to add the tombstones for all successful deletes, 
* createFakeDirectory to handle failure on the GET or the PUT
* maybe also: add specific metric for permissions failures 

rename() is what worries me the most with s3guard, as its the one which will end up most inconsistent.

Everything is setup to add tests for these; harder part is fixing. Thank you for volunteering

> s3a: clarify / improve support for mixed ACL buckets
> ----------------------------------------------------
>                 Key: HADOOP-15525
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15525
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 3.0.0
>            Reporter: Aaron Fabbri
>            Assignee: Aaron Fabbri
>            Priority: Major
> Scenario: customer wants to only give a Hadoop cluster access to a subtree of an S3
> For example, assume Hadoop uses some IAM identity "hadoop", which they wish to grant
full permission to everything under the following path:
> s3a://bucket/a/b/c/hadoop-dir
> they don't want hadoop user to be able to read/list/delete anything outside of the hadoop-dir
> Problems: 
> To implement the "directory structure on flat key space" emulation logic we use to present
a Hadoop FS on top of a blob store, we need to create / delete / list ancestors of {{hadoop-dir}}.
(to maintain the invariants (1) zero-byte object with key ending in '/' exists iff empty directory
is there and (2) files cannot live beneath files, only directories.)
> I'd like us to (1) document a an example with IAM ACLs policies that gets this basic
functionality, and consider (2) making improvements to make this easier.
> We've discussed some of these issues before but I didn't see a dedicated JIRA.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message