hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11038) Support AWS roles to talk to AWS-S3 largely for cross-AWS-account integration
Date Fri, 06 Feb 2015 21:41:35 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309978#comment-14309978

Steve Loughran commented on HADOOP-11038:


We aren't going to go anywhere near patching jets3t. 

Do that and 
# it breaks the whole maven build process. There is no jets3t in the  hadoop tree
# we would be taking on all the support and maintenance obligations of that JAR.

The new s3a filesystem client is built using the AWS toolkit, not jets3t. Please see if it
can support this feature

> Support AWS roles to talk to AWS-S3 largely for cross-AWS-account integration
> -----------------------------------------------------------------------------
>                 Key: HADOOP-11038
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11038
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs, fs/s3
>            Reporter: Vishal Gupta
>         Attachments: HADOOP-11038.1.patch, HADOOP-11038.2.patch
> Currently "hdfs dfs -lsr s3://..." supports acess-keys/secret-keys only as the way to
authenticate to s3. This should support AWS-roles also because of the following reasons :
> 1) AWS-roles is a AWS best-practice and is highly recommended by AWS themselves.
> 2) This helps in cross-AWS-account integration also. An AWS-account-holder can provide
another AWS-account-holder a cross-account-AWS-role to perform operations over his S3-buckets.
> The current syntax is "hdfs dfs" is :
> hdfs  dfs  -Dfs.s3n.awsAccessKeyId=XXXX -Dfs.s3n.awsSecretAccessKey=XXXX -ls  s3n://.../
> This should change to :
> hdfs dfs  -Dfs.s3n.awsAccessKeyId=XXXX -Dfs.s3n.awsSecretAccessKey=XXXX -Dfs.s3n.awsRoleToBeAssumed=arn:aws:iam::XXXX:role/XXXX
-Dfs.s3n.awsExternalId=XXXX -ls s3n://.../
> Extending the use-case a little further, for a client AWS-account to integrate with multiple
different AWS-accounts, configuration for s3-bucket to role-to-be-assumed mapping ( which
will override the master-role ) should be provided :
> hdfs  dfs  -Dfs.s3.awsAccessKeyId=XXXX -Dfs.s3.awsSecretAccessKey=XXXX -Dfs.s3.awsRoleToBeAssumed=arn:aws:iam::XXXX:role/XXXX
-Dfs.s3.awsBucketToRoleMapping="{\"bucket1\": { \"roleName\":\"arn:aws:iam::XXXX:role/role1\",
\"externalId\":\"....\"}}" -ls s3://.../
> Since, AWS treats a cross-account-AWS-role the same as an AWS-role within a AWS-account,
the above flows remain same for a role within a AWS-account.

This message was sent by Atlassian JIRA

View raw message