hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-14556) S3A to support Delegation Tokens
Date Mon, 15 Oct 2018 21:52:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650850#comment-16650850
] 

Steve Loughran edited comment on HADOOP-14556 at 10/15/18 9:51 PM:
-------------------------------------------------------------------

Note that the partially {{ITestDelegatedMRJob}} test does show that S3A tokens are picked
up for MR job submit; tested for full, session and role tokens.

One fun detail: if your fs.s3a.secret.key &c attributes are set in the job conf you launch
with, they end up at the far end, even though you are using DTs. Why? well, because they are
config options, aren't they?

To get the lockdown to work, you need to be serving up the secrets inside a hadoop credential
provider file such as  localjceks file. That way, the job conf will not contain the secrets.

There's no obvious way to patch the options, so that's going to have to go down as what to
do. Setting the AWS env vars would also work, though as spark automatically picks up those
values and patches the fs config (without any check for the properties first), they may get
in.


was (Author: stevel@apache.org):
Note that the partially {{ITestDelegatedMRJob}} test does show that S3A tokens are picked
up for MR job submit; tested for full, session and role tokens.

One fun detail: if your fs.s3a.secret.key &c attributes are set in the job conf you launch
with, they end up at the far end, even though you are using DTs. Why? well, because they are
config options, aren't they?

To get the lockdown to work, you need to be serving up the secrets inside a hadoop credential
provider file such as  localjceks file. That way, the job conf will not contain the secrets.
There's no 

> S3A to support Delegation Tokens
> --------------------------------
>
>                 Key: HADOOP-14556
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14556
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.2.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>         Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, HADOOP-14556-003.patch,
HADOOP-14556-004.patch, HADOOP-14556-005.patch, HADOOP-14556-007.patch, HADOOP-14556-008.patch,
HADOOP-14556-009.patch, HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch,
HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id; these
will be saved in the token and  marshalled with jobs
> * A new authentication provider will look for a token for the current user and authenticate
the user if found
> This will not support renewals; the lifespan of a token will be limited to the initial
duration. Also, as you can't request an STS token from a temporary session, IAM instances
won't be able to issue tokens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message