hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Alten-Lorenz <wget.n...@gmail.com>
Subject Re: Transferring security tokens to remote machines
Date Thu, 12 Feb 2015 17:19:07 GMT
Hi Robert,

forgive me if I’m wrong, but so far as I understand Flink uses nearly the same model as
HDFS (not at all). Means the master receives an action and distribute that to the workers
(more or less ;)) 
HDFS as example uses not an push mechanism, the DN clients fetch the token from the NN when
they need them. Could that be a solution, too?

https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
<https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java>

MapReduce gets the token from the JT with getDelegationToken()

http://hadoop.apache.org/docs/r2.5.2/api/org/apache/hadoop/mapreduce/Cluster.html#getDelegationToken(org.apache.hadoop.io.Text)
<http://hadoop.apache.org/docs/r2.5.2/api/org/apache/hadoop/mapreduce/Cluster.html#getDelegationToken(org.apache.hadoop.io.Text)>


BR,
 Alexander 


> On 12 Feb 2015, at 15:28, Robert Metzger <rmetzger@apache.org> wrote:
> 
> Hi,
> 
> I'm a committer at the Apache Flink project.
> One of our users asked for adding support for reading from a secured HDFS cluster.
> 
> Flink has a master-worker model. Since its not really feasible for users to login with
their kerberos credentials on all workers, I wanted to acquire the security token on the master
and send it to all workers.
> For that, I wrote the following code to get the tokens in to a byte array:
> 
> UserGroupInformation.setConfiguration(hdConf);
> Credentials credentials = new Credentials();
> UserGroupInformation currUsr = UserGroupInformation.getCurrentUser();
> 
> Collection<Token<? extends TokenIdentifier>> usrTok = currUsr.getTokens();
> for(Token<? extends TokenIdentifier> token : usrTok) {
>    final Text id = new Text(token.getIdentifier());
>    credentials.addToken(id, token);
> }
> DataOutputBuffer dob = new DataOutputBuffer();
> credentials.writeTokenStorageToStream(dob);
> dob.flush();
> However, the collection currUsr.getTokens() is empty, hence the output buffer doesn't
contain much data.
> I suspect that I didn't fully understand the Hadoop security concepts yet.
> It would be great if somebody from the list could clarify how to properly acquire the
tokens.
> 
> Also, I was wondering if there is any document describing how the UserGroupInformation
class is working (when is it loading the credentials, does it only work for Kerberos, ...)
> 
> Best,
> Robert
> 


Mime
View raw message