hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13252) add logging of what's going on in s3 auth to help debug problems
Date Thu, 09 Jun 2016 20:18:21 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323261#comment-15323261

Chris Nauroth commented on HADOOP-13252:

One thing that's helpful, and works now without code changes, is to add this to log4j.properties:


Then the AWS SDK will print which AWS credential provider was selected from the chain, without
leaking any secrets.

> hadoop fs -ls s3a://cnauroth-test-aws-s3a/
16/06/09 13:14:13 DEBUG auth.AWSCredentialsProviderChain: Loading credentials from BasicAWSCredentialsProvider

However, if this is more about Hadoop's credential provider API, then I don't think it has
the logging you're looking for yet.

> add logging of what's going on in s3 auth to help debug problems
> ----------------------------------------------------------------
>                 Key: HADOOP-13252
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13252
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.8.0
>            Reporter: Steve Loughran
>            Priority: Minor
> We've now got some fairly complex auth mechanisms going on: -hadoop config, KMS, env
vars, "none". IF something isn't working, it's going to be a lot harder to debug.
> I propose *carefully* adding some debug messages to identify which auth provider is doing
the auth, so we can see if the env vars were kicking in, sysprops, etc.
> What we mustn't do is leak any secrets: this should be identifying whether properties
and env vars are set, not what their values are. I don't believe that this will generate a
security risk.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message