hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Transferring security tokens to remote machines
Date Thu, 12 Feb 2015 17:55:41 GMT
Hi,

thank you for the quick reply. I'll look into the links to see if we can
implement a similar mechanism.

Robert

On Thu, Feb 12, 2015 at 6:19 PM, Alexander Alten-Lorenz <wget.null@gmail.com
> wrote:

> Hi Robert,
>
> forgive me if I’m wrong, but so far as I understand Flink uses nearly the
> same model as HDFS (not at all). Means the master receives an action and
> distribute that to the workers (more or less ;))
> HDFS as example uses not an push mechanism, the DN clients fetch the token
> from the NN when they need them. Could that be a solution, too?
>
>
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
>
> MapReduce gets the token from the JT with getDelegationToken()
>
>
> http://hadoop.apache.org/docs/r2.5.2/api/org/apache/hadoop/mapreduce/Cluster.html#getDelegationToken(org.apache.hadoop.io.Text)
>
>
> BR,
>  Alexander
>
>
> On 12 Feb 2015, at 15:28, Robert Metzger <rmetzger@apache.org> wrote:
>
> Hi,
>
> I'm a committer at the Apache Flink project.
> One of our users asked for adding support for reading from a secured HDFS
> cluster.
>
> Flink has a master-worker model. Since its not really feasible for users
> to login with their kerberos credentials on all workers, I wanted to
> acquire the security token on the master and send it to all workers.
> For that, I wrote the following code to get the tokens in to a byte array:
>
> UserGroupInformation.setConfiguration(hdConf);
> Credentials credentials = new Credentials();
> UserGroupInformation currUsr = UserGroupInformation.getCurrentUser();
>
> Collection<Token<? extends TokenIdentifier>> usrTok = currUsr.getTokens();
> for(Token<? extends TokenIdentifier> token : usrTok) {
>    final Text id = new Text(token.getIdentifier());
>    credentials.addToken(id, token);
> }
> DataOutputBuffer dob = new DataOutputBuffer();
> credentials.writeTokenStorageToStream(dob);
> dob.flush();
>
> However, the collection currUsr.getTokens() is empty, hence the output buffer doesn't
contain much data.
>
> I suspect that I didn't fully understand the Hadoop security concepts yet.
> It would be great if somebody from the list could clarify how to properly acquire the
tokens.
>
> Also, I was wondering if there is any document describing how the UserGroupInformation
class is working (when is it loading the credentials, does it only work for Kerberos, ...)
>
>
> Best,
>
> Robert
>
>
>
>

Mime
View raw message