hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12723) S3A: Add ability to plug in any AWSCredentialsProvider
Date Wed, 18 May 2016 12:58:12 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15288907#comment-15288907

Steve Loughran commented on HADOOP-12723:

# The patch reverts a couple of things in the toString() call. I'll go with line 1395, but
not sure about the L1383 change; the existing code omits the line completely.
# {{getAWSCredentialsProvider}} : need to have consensus there on what classloader to use
# Once HADOOP-13130 gets in then it'll break {{TestS3AAWSCredentialsProvider}}; all Amazon
exceptions will be caught and wrapped by IOEs; your test will need to change. It'll be 
a race to see whose gets in first...
# {{TestS3AAWSCredentialsProvider}}: codepaths that expected to raise an exception should
have a {{fail(message)}} on the path which is expected to fail. Otherwise test failures could
be missed
# you'll need to cover the new properties in the AWS {{site/index.md}} file.

One question I have here is how would this relate to HADOOP-12537; STS support? I believe
That mandates the STS JAR on the CP, and adds extra config options for the binding. If it's
the latter, the binding info, this means that your patch isn't going to be sufficient on its
own. What would it take to support it? 

Presumably, it'd need a new provider which looked up the session key from the properties,
yes? If so, that'd be the first one to add after this patch, marking the STS JAR as  a "provided"
dependency of hadoop-aws. 

What other providers do you have in mind?

> S3A: Add ability to plug in any AWSCredentialsProvider
> ------------------------------------------------------
>                 Key: HADOOP-12723
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12723
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/s3
>    Affects Versions: 2.7.1
>            Reporter: Steven Wong
>            Assignee: Steven Wong
>         Attachments: HADOOP-12723.0.patch, HADOOP-12723.1.patch, HADOOP-12723.2.patch,
HADOOP-12723.3.patch, HADOOP-12723.4.patch
> Although S3A currently has built-in support for {{org.apache.hadoop.fs.s3a.BasicAWSCredentialsProvider}},
{{com.amazonaws.auth.InstanceProfileCredentialsProvider}}, and {{org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider}},
it does not support any other credentials provider that implements the {{com.amazonaws.auth.AWSCredentialsProvider}}
interface. Supporting the ability to plug in any {{com.amazonaws.auth.AWSCredentialsProvider}}
instance will expand the options for S3 credentials, such as:
> * temporary credentials from STS, e.g. via {{com.amazonaws.auth.STSSessionCredentialsProvider}}
> * IAM role-based credentials, e.g. via {{com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider}}
> * a custom credentials provider that satisfies one's own needs, e.g. bucket-specific
credentials, user-specific credentials, etc.
> To support this, we can add a configuration for the fully qualified class name of a credentials
provider, to be loaded by {{S3AFileSystem.initialize(URI, Configuration)}}.
> The configured credentials provider should implement {{com.amazonaws.auth.AWSCredentialsProvider}}
and have a constructor that accepts {{(URI uri, Configuration conf)}}.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message