hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
Date Mon, 27 Mar 2017 11:42:41 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943104#comment-15943104

Steve Loughran commented on HADOOP-14237:

+persisted data structure should use the JECKS encrypted credential mechanism, so that it
isn't stored in plaintext, even in HDFS. Processes which can access the data would need to
be given the shared key and path needed to find and read the data,

> S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
> -----------------------------------------------------------------------
>                 Key: HADOOP-14237
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14237
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1
>         Environment: EC2, AWS
>            Reporter: Kazuyuki Tanimura
> When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails getting the
instance profile credentials, eventually all jobs on the cluster fail. Since a number of S3A
clients (all mappers and reducers) try to get the credentials, the AWS credential endpoint
starts responding 5xx and 4xx error codes.
> SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, but it still
does not share the credentials with other EC2 nodes / JVM processes.
> This issue prevents users from creating Hadoop clusters on EC2

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message