hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12806) Hadoop fs s3a lib not working with temporary credentials in AWS Lambda
Date Wed, 15 Feb 2017 10:57:41 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15867640#comment-15867640
] 

Steve Loughran commented on HADOOP-12806:
-----------------------------------------

Given Hadoop 2.8 lets you declare whatever provider you want, it should be possible to wire
things up, even if we aren't doing it out the box in a way which works with AWS Lambda.

Nikolaos: could you build the latest 2.8 branch and see if it works now?

> Hadoop fs s3a lib not working with temporary credentials in AWS Lambda 
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-12806
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12806
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>    Affects Versions: 2.7.2
>            Reporter: Nikolaos Tsipas
>              Labels: aws-s3
>
> Trying to use the hadoop fs s3a library in AWS lambda with temporary credentials but
it's not possible because of the way the {{AWSCredentialsProviderChain}} is defined under
https://github.com/apache/hadoop/blob/29ae25801380b94442253c4202dee782dc4713f5/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
> Specifically the following code is used to initialise the credentials chain
> {code}
> AWSCredentialsProviderChain credentials = new AWSCredentialsProviderChain(
>         new BasicAWSCredentialsProvider(accessKey, secretKey),
>         new InstanceProfileCredentialsProvider(),
>         new AnonymousAWSCredentialsProvider()
>     );
> {code}
> The above works fine when the EC2 metadata endpoint is available (i.e. running on an
EC2 instance) however it doesn't work properly when the environment variables  are used to
define credentials as it happens in AWS Lambda. Amazon suggests to use the {{EnvironmentVariableCredentialsProvider}}
in AWS Lambda. 
> To summarise and suggest an alternative I think that the {{DefaultAWSCredentialsProviderChain}}
could be used instead of  the {{InstanceProfileCredentialsProvider}}  and that would cover
the following cases: 
> {panel}
> * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (RECOMMENDED since
they are recognized by all the AWS SDKs and CLI except for .NET), or AWS_ACCESS_KEY and AWS_SECRET_KEY
(only recognized by Java SDK)
> * Java System Properties - aws.accessKeyId and aws.secretKey
> * Credential profiles file at the default location (~/.aws/credentials) shared by all
AWS SDKs and the AWS CLI
> * Instance profile credentials delivered through the Amazon EC2 metadata service
> {panel}
> If you think that the above change would be useful I could investigate more about what
the required changes would be and submit a patch.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message