hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Larry McCay (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
Date Sun, 30 Dec 2018 20:09:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16731063#comment-16731063

Larry McCay commented on HADOOP-14556:

Hi [~stevel@apache.org] - I have finally gotten through the majority of the patch.

This looks like a great contribution!

A few observations that I would like to verify:
 # If delegation tokens are enabled it seems that there is no fallback to other providers.
This seems appropriate to me but just want to make sure that interpretation is correct.
 # While there are specific implementations provided in the patch it also seems to be extensible
by 3rd parties that which to provide their own DT and binding code.

Can you provide any details on renewal of DT's in the provided implementations?

A number of nits and questions/comments from the review:

Typos: s/provide/to provide/


+ The name of a class provide delegation tokens support in S3A.
+ If unset: delegation token support is disnabled.


Probably want a space after name in 


+ String message = name + "No AWS Credentials provided by "
Actually the javadoc seems to indicate that ": " will always be there.
+ /**
+ * The name, with a ": " suffix.
+ */
+ private String name = "";


How is this policed and does it really make sense in a name?

Can you explain what the name is for here?
Are we providing the ability to have multiple named provider chains in a config?

Is the following supposed to be an empty string and the ASSUMED_ROLE_STS_ENDPOINT_REGION would
be set if DEFAULT_ASSUMED_ROLE_STS_ENDPOINT isn't an empty string?


+ * Default endpoint for session tokens: \{@value}.
+ * This is the central STS endpoint which, for v3 signing, can
+ * issue STS tokens for any region.
+ */
+ public static final String DEFAULT_ASSUMED_ROLE_STS_ENDPOINT = "";
+ /**
+ * Region for the STS endpoint; needed if the endpoint
+ * is set to anything other then the central one.: \{@value}.
 public static final String ASSUMED_ROLE_STS_ENDPOINT_REGION =


and again


+ public static final String ASSUMED_ROLE_STS_ENDPOINT_REGION_DEFAULT = "";




Is it possible for the toString() to have credentials from the URI?


+ @Override
+ public String toString() {
+ final StringBuilder sb = new StringBuilder("S3A{");
+ sb.append("URI =").append(fsImpl.getUri());
+ sb.append("; fsImpl=").append(fsImpl);
+ sb.append('}');
+ return sb.toString();


logAtInfo - should this be javadoc'd to explain how and when it is set to debug?

Can the following javadoc be better explained?


 * There's scope in here for client encryption options, even while not
 * currently supported in S3A.


Also, should we consider making encryption algorithms such as AES256 configurable rather than


> S3A to support Delegation Tokens
> --------------------------------
>                 Key: HADOOP-14556
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14556
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.3.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>         Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, HADOOP-14556-003.patch,
HADOOP-14556-004.patch, HADOOP-14556-005.patch, HADOOP-14556-007.patch, HADOOP-14556-008.patch,
HADOOP-14556-009.patch, HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch,
HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, HADOOP-14556-015.patch,
HADOOP-14556-016.patch, HADOOP-14556-017.patch, HADOOP-14556-018a.patch, HADOOP-14556-019.patch,
HADOOP-14556-020.patch, HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch,
HADOOP-14556-024.patch, HADOOP-14556-025.patch, HADOOP-14556-026.patch, HADOOP-14556.oath-002.patch,
> S3A to support delegation tokens where
> * an authenticated client can request a token via {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id; these
will be saved in the token and  marshalled with jobs
> * A new authentication provider will look for a token for the current user and authenticate
the user if found
> This will not support renewals; the lifespan of a token will be limited to the initial
duration. Also, as you can't request an STS token from a temporary session, IAM instances
won't be able to issue tokens.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message