hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
Date Fri, 02 Jun 2017 19:49:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035306#comment-16035306
] 

Daryn Sharp commented on HADOOP-14445:
--------------------------------------

I had a feeling "nameservice" alluded to the hdfs HA configuration – which is horrible
for the reasons I've detailed and why we don't use it.  I'll politely stress and repeat: *Updating
configs of tens of thousands of nodes, launchers, oozie, storm, spark, etc and restarting
the services is just not logistically possible*.

bq. Although, IIRC, the tokens are renewed only if they are expired, so if they are renewed
serially, it should not be a problem.

The RM renews immediately to verify token validity and to determine the next renewal time.
 If they are expired, it's too late.  Any kp token using just a service authority cannot determine
the kp uri and is only renewable via the kp uri in the config – enforcing one and only
1 kms cluster.  If the kp client can be instantiated via the service, then multi-kms setups
are possible.

bq.  I do like the idea of using a nameservice though, as Yongjun Zhang suggested which will
ensure that we will still have only 1 single entry.

There must be a disconnect here.  1 single entry is the advantage of just setting the service
to the provider uri.  Adding an extra layer of indirection through the config creates a logistical
mess with no added benefits.  I'm not going to bounce all my services and RMs because I added
or changed a KMS cluster.

Here's the big picture we are trying to achieve:
* client requests kp uri from NN
* client creates kp client from kp uri
* client gets tokens and sets service to kp uri
* RM calls kms token renewer which uses kp uri in service to create kp client
* tasks use the NN->kp uri mapping established at job submission to locate tokens

It's +config-less+ other than a setting on the NN.  This is what we are running internally
because the current kms client design is completely broken.  We now have the ability to enable
EZs on a NN and/or change kms cluster configuration without changing configs or restarting
services.

We only care about this load balancing provider because we need to ensure the kp client can
be instantiated from the service.

> Delegation tokens are not shared between KMS instances
> ------------------------------------------------------
>
>                 Key: HADOOP-14445
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14445
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation, kms
>    Affects Versions: 2.8.0, 3.0.0-alpha1
>            Reporter: Wei-Chiu Chuang
>            Assignee: Rushabh S Shah
>         Attachments: HADOOP-14445-branch-2.8.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do not share
delegation tokens. (a client uses KMS address/port as the key for delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
>         InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
>             url.getPort());
>         Text service = SecurityUtil.buildTokenService(serviceAddr);
>         dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation tokens
too.
> Under HA, A KMS instance must verify the delegation token given by another KMS instance,
by checking the shared secret used to sign the delegation token. To do this, all KMS instances
must be able to retrieve the shared secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share delegation tokens.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message