Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7EA18101BD for ; Thu, 12 Feb 2015 17:22:03 +0000 (UTC) Received: (qmail 71477 invoked by uid 500); 12 Feb 2015 17:21:48 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 71370 invoked by uid 500); 12 Feb 2015 17:21:48 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 71360 invoked by uid 99); 12 Feb 2015 17:21:48 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Feb 2015 17:21:48 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of wget.null@gmail.com designates 74.125.82.175 as permitted sender) Received: from [74.125.82.175] (HELO mail-we0-f175.google.com) (74.125.82.175) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Feb 2015 17:21:20 +0000 Received: by mail-we0-f175.google.com with SMTP id x3so11548983wes.6 for ; Thu, 12 Feb 2015 09:19:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:message-id:mime-version:subject:date:references :to:in-reply-to; bh=QQj7QJn7B7gV2oTW5uFkoMeC1bLHhICPA8uaZM0nMs0=; b=TzBZhBqtJGh7GI/NArGToEwv2Jnaj7Dk8cIu8azbQ7+fJFEZZMHN/PT3/d6oni22DY 9iZsGpA6xaDHsFLjknar88O949X1TB/kGjm+hMcFDIWddaqTmkTn07/wtXWsF8vfodls ULFHTe/m13xSHfzIpAE3V6NaAytDjkVXagkFydcFEcMS9eshRU6sh1jC78xGfkr5cs46 k8oIt+jqRaqMYGfNRtxsxKBKQ+woivLG144VWD6BQwiL8yTFyGOPNdlBLA1JtoqWrDDz unkPRDSNRsbNBf6ZGsKbMipeaZHClrAe7vIlLsptZMAuIF4GpzyPyJle8lyLQ6ez5mc5 hRQQ== X-Received: by 10.180.198.148 with SMTP id jc20mr8209046wic.67.1423761589432; Thu, 12 Feb 2015 09:19:49 -0800 (PST) Received: from donald.railnet.train ([88.128.80.17]) by mx.google.com with ESMTPSA id l6sm6482952wjx.33.2015.02.12.09.19.39 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 12 Feb 2015 09:19:48 -0800 (PST) From: Alexander Alten-Lorenz Content-Type: multipart/alternative; boundary="Apple-Mail=_EFC0DF7E-66EB-4CD1-8A5F-957805FB580E" Message-Id: <13C6FFF8-2746-4B29-BFF2-86A58BCA1949@gmail.com> Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) Subject: Re: Transferring security tokens to remote machines Date: Thu, 12 Feb 2015 18:19:07 +0100 References: To: user@hadoop.apache.org In-Reply-To: X-Mailer: Apple Mail (2.2070.6) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_EFC0DF7E-66EB-4CD1-8A5F-957805FB580E Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Robert, forgive me if I=E2=80=99m wrong, but so far as I understand Flink uses = nearly the same model as HDFS (not at all). Means the master receives an = action and distribute that to the workers (more or less ;))=20 HDFS as example uses not an push mechanism, the DN clients fetch the = token from the NN when they need them. Could that be a solution, too? = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1/src/hdfs/= org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java = MapReduce gets the token from the JT with getDelegationToken() = http://hadoop.apache.org/docs/r2.5.2/api/org/apache/hadoop/mapreduce/Clust= er.html#getDelegationToken(org.apache.hadoop.io.Text) = BR, Alexander=20 > On 12 Feb 2015, at 15:28, Robert Metzger wrote: >=20 > Hi, >=20 > I'm a committer at the Apache Flink project. > One of our users asked for adding support for reading from a secured = HDFS cluster. >=20 > Flink has a master-worker model. Since its not really feasible for = users to login with their kerberos credentials on all workers, I wanted = to acquire the security token on the master and send it to all workers. > For that, I wrote the following code to get the tokens in to a byte = array: >=20 > UserGroupInformation.setConfiguration(hdConf); > Credentials credentials =3D new Credentials(); > UserGroupInformation currUsr =3D = UserGroupInformation.getCurrentUser(); >=20 > Collection> usrTok =3D = currUsr.getTokens(); > for(Token token : usrTok) { > final Text id =3D new Text(token.getIdentifier()); > credentials.addToken(id, token); > } > DataOutputBuffer dob =3D new DataOutputBuffer(); > credentials.writeTokenStorageToStream(dob); > dob.flush(); > However, the collection currUsr.getTokens() is empty, hence the output = buffer doesn't contain much data. > I suspect that I didn't fully understand the Hadoop security concepts = yet. > It would be great if somebody from the list could clarify how to = properly acquire the tokens. >=20 > Also, I was wondering if there is any document describing how the = UserGroupInformation class is working (when is it loading the = credentials, does it only work for Kerberos, ...) >=20 > Best, > Robert >=20 --Apple-Mail=_EFC0DF7E-66EB-4CD1-8A5F-957805FB580E Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
Hi Robert,

forgive me if I=E2=80=99m wrong, but so = far as I understand Flink uses nearly the same model as HDFS (not at = all). Means the master receives an action and distribute that to the = workers (more or less ;)) 
HDFS as example = uses not an push mechanism, the DN clients fetch the token from the NN = when they need them. Could that be a solution, too?

https://svn.apache.org/repos/asf/hadoop/common/branches/branch-= 1/src/hdfs/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java

MapReduce gets = the token from the JT with getDelegationToken()

http://hadoop.apache.org/docs/r2.5.2/api/org/apache/hadoop/mapr= educe/Cluster.html#getDelegationToken(org.apache.hadoop.io.Text)
=


BR,
 Alexander 


On 12 Feb 2015, at 15:28, Robert Metzger = <rmetzger@apache.org> wrote:

Hi,

I'm a = committer at the Apache Flink project.
One of our = users asked for adding support for reading from a secured HDFS = cluster.

Flink = has a master-worker model. Since its not really feasible for users to = login with their kerberos credentials on all workers, I wanted to = acquire the security token on the master and send it to all = workers.
For that, I wrote the following code to = get the tokens in to a byte array:

UserGroupInformation.setConfiguration(hdConf);
Credentials credentials =3D =
new =
Credentials();
UserGroupInformation currUsr =3D =
UserGroupInformation.getCurrentUser();

Collection<Token<? extends =
TokenIdentifier>> usrTok =3D currUsr.getTokens();
for(Token<? extends =
TokenIdentifier> token : usrTok) {
   final =
Text id =3D new Text(token.getIdentifier());
   =
credentials.addToken(id, token);
}
DataOutputBuffer dob =3D new =
DataOutputBuffer();
credentials.writeTokenStorageToStream(dob);
dob.flush();
However, the collection currUsr.getTokens() =
is empty, =
hence the output buffer doesn't contain much data.

I suspect that I didn't fully understand the Hadoop security =
concepts yet.
It would be great if somebody from the list could clarify how to =
properly acquire the tokens.

Also, I was wondering if there is any document describing how the =
UserGroupInformation class is working (when is it loading the =
credentials, does it only work for Kerberos, ...)

Best,
Robert


= --Apple-Mail=_EFC0DF7E-66EB-4CD1-8A5F-957805FB580E--