hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-14556) S3A to support Delegation Tokens
Date Mon, 15 Oct 2018 21:46:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650831#comment-16650831
] 

Steve Loughran edited comment on HADOOP-14556 at 10/15/18 9:45 PM:
-------------------------------------------------------------------

HADOOP-14556 patch 013
* ITestDelegatedMRJob mixes a mock job submission API with a real miniYarn cluster to verify
that MR job submission collects DTs for source and destination paths.
  To do this the MockJob class had to go into hadoop-aws/src/test/java/org/apache/hadoop/mapreduce/MockJob.java
and job.connect() made an override point (so it can be skipped)
* default assumed role duration returned to 1h; it had been extended to 6h but that only works
if your role has been explicitly extended to > 1h duration.
* and docs on increasing it (plus error messages you get if you don't) improved/extended in
assumed_roles.md as well as delegation_tokens.md.
 All AWS error messages related to STS/session and role requests are now in assumed_roles.md
to avoid duplication & inconsistencies.
* ITestS3ADelegationTokenSupport tests that the Session DT binding will forward any session
creds it gets from its own auth chain, rather than ask for new ones (which it can't do with
session creds)
* Also: I'm using a Hadoop cred provider for storing secrets; this broke the AssumeRole and
delegation tests which were clearing or overwriting the fs.s3a.{auth, secret, session} options,
as those in the creds file were still being picked up. Fix: explicitly reset hadoop.security.credential.provider.path
for all the tests which were now failing.
* minor checkstyle fixup

tested, S3A ireland. Apart from the cred problem (fixed), I got a failure of {{ITestS3GuardToolLocal\#testDestroyNoBucket}}
*even when I was running with dynamodb*. I think that test suite is running when it shouldn't.
More research needed there


was (Author: stevel@apache.org):
HADOOP-14556 patch 013
* ITestDelegatedMRJob mixes a mock job submission API with a real miniYarn cluster to verify
that MR job submission collects DTs for source and destination paths.
  To do this the MockJob class had to go into hadoop-aws/src/test/java/org/apache/hadoop/mapreduce/MockJob.java
and job.connect() made an override point (so it can be skipped)
* default assumed role duration returned to 1h; it had been extended to 6h but that only works
if your role has been explicitly extended to > 1h duration.
* and docs on increasing it (plus error messages you get if you don't) improved/extended in
assumed_roles.md as well as delegation_tokens.md.
 All AWS error messages related to STS/session and role requests are now in assumed_roles.md
to avoid duplication & inconsistencies.
* ITestS3ADelegationTokenSupport tests that the Session DT binding will forward any session
creds it gets from its own auth chain, rather than ask for new ones (which it can't do with
session creds)
* Also: I'm using a Hadoop cred provider for storing secrets; this broke the AssumeRole and
delegation tests which were clearing or overwriting the fs.s3a.{auth, secret, session} options,
as those in the creds file were still being picked up. Fix: explicitly reset hadoop.security.credential.provider.path
for all the tests which were now failing.
* minor checkstyle fixup

tested, S3A ireland. Apart from the cred problem (fixed), I got a failure of {{ITestS3GuardToolLocal\#testDestroyNoBucket
}} *even when I was running with dynamodb*. I think that test suite is running when it shouldn't.
More research needed there

> S3A to support Delegation Tokens
> --------------------------------
>
>                 Key: HADOOP-14556
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14556
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.2.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>         Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, HADOOP-14556-003.patch,
HADOOP-14556-004.patch, HADOOP-14556-005.patch, HADOOP-14556-007.patch, HADOOP-14556-008.patch,
HADOOP-14556-009.patch, HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch,
HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id; these
will be saved in the token and  marshalled with jobs
> * A new authentication provider will look for a token for the current user and authenticate
the user if found
> This will not support renewals; the lifespan of a token will be limited to the initial
duration. Also, as you can't request an STS token from a temporary session, IAM instances
won't be able to issue tokens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message