hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kazuyuki Tanimura (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
Date Sat, 25 Mar 2017 02:30:41 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Kazuyuki Tanimura updated HADOOP-14237:
---------------------------------------
    Description: 
When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails getting the instance
profile credentials, eventually all jobs on the cluster fail. Since a number of S3A clients
(all mappers and reducers) try to get the credentials, the AWS credential endpoint starts
responding 5xx and 4xx error codes.

SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, but it still
does not share the credentials with other EC2 nodes / JVM processes.

This issue prevents users from creating Hadoop clusters on EC2

  was:
When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails getting the instance
profile credentials, eventually all jobs on the cluster fail. Since a number of S3A clients
(all mappers and reducers) try to get the credentials, the AWS credential endpoint starts
responding 5xx and 4xx error codes.

SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, but it still
does not share the credentials with other EC2 nodes / processes.

This issue prevents users from creating Hadoop clusters on EC2


> S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-14237
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14237
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1
>         Environment: EC2, AWS
>            Reporter: Kazuyuki Tanimura
>
> When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails getting the
instance profile credentials, eventually all jobs on the cluster fail. Since a number of S3A
clients (all mappers and reducers) try to get the credentials, the AWS credential endpoint
starts responding 5xx and 4xx error codes.
> SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, but it still
does not share the credentials with other EC2 nodes / JVM processes.
> This issue prevents users from creating Hadoop clusters on EC2



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message