Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E68D8181C2 for ; Wed, 19 Aug 2015 22:47:56 +0000 (UTC) Received: (qmail 31401 invoked by uid 500); 19 Aug 2015 22:47:51 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 31301 invoked by uid 500); 19 Aug 2015 22:47:51 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 31280 invoked by uid 99); 19 Aug 2015 22:47:51 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2015 22:47:51 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 0FD58182105 for ; Wed, 19 Aug 2015 22:47:51 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.999 X-Spam-Level: ** X-Spam-Status: No, score=2.999 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id AjWsQHHPCurw for ; Wed, 19 Aug 2015 22:47:48 +0000 (UTC) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2on0091.outbound.protection.outlook.com [207.46.100.91]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id CCDC820756 for ; Wed, 19 Aug 2015 22:47:47 +0000 (UTC) Received: from SN1PR0701MB1837.namprd07.prod.outlook.com (10.162.100.154) by SN1PR0701MB1837.namprd07.prod.outlook.com (10.162.100.154) with Microsoft SMTP Server (TLS) id 15.1.231.21; Wed, 19 Aug 2015 22:47:40 +0000 Received: from SN1PR0701MB1837.namprd07.prod.outlook.com ([10.162.100.154]) by SN1PR0701MB1837.namprd07.prod.outlook.com ([10.162.100.154]) with mapi id 15.01.0231.024; Wed, 19 Aug 2015 22:47:40 +0000 From: John Lilley To: "user@hadoop.apache.org" Subject: RE: UserGroupInformation and login with password Thread-Topic: UserGroupInformation and login with password Thread-Index: AdDZAMDabfxkIEHfSUSGx+g+Y64pCwAQ4ZHQAGMjFZA= Date: Wed, 19 Aug 2015 22:47:39 +0000 Message-ID: References: <8D5F7E3237B3ED47B84CF187BB17B66611C0BFEA@SHSMSX103.ccr.corp.intel.com> In-Reply-To: <8D5F7E3237B3ED47B84CF187BB17B66611C0BFEA@SHSMSX103.ccr.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=john.lilley@redpoint.net; x-originating-ip: [173.160.43.60] x-microsoft-exchange-diagnostics: 1;SN1PR0701MB1837;5:EZ3lXHrujKCFzmneTk2B1OLB+XEdX1H486/e2LY6ydZxjuno1hr287CcKsOE2Mgm9BlpjlbLbYpv5QdfutdkmkPp5Q8/r7FMn6kMDcPDzA9peG7LyrgD30eJoh1yjPyOhRpRdSGTcLyB5FzqCgFoKw==;24:a6bw7iEJwoKOjsI/42QbZWQtDvZ3fj45ZovuqGrb3YmyH5bfzJE7boThsGycvJzG5c4f/67aFRswCwzi3zwb3RCkmB3YI2nxubqYirOnJhY=;20:ZCT2NuNz4bMfV8BS2vrTxxt7GMCrbSbG0S2dGk7u3GZSguhQMShETo/qaXRPvgJYyvB+wn2AEGq+ZTuTt2wEfg== x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:SN1PR0701MB1837; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(108003899814671); x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(601004)(8121501046)(5005006)(3002001);SRVR:SN1PR0701MB1837;BCL:0;PCL:0;RULEID:;SRVR:SN1PR0701MB1837; x-forefront-prvs: 0673F5BE31 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(377454003)(199003)(129404003)(189002)(450100001)(16236675004)(46102003)(87936001)(101416001)(77096005)(2501003)(68736005)(62966003)(76576001)(19625215002)(77156002)(54356999)(19300405004)(2950100001)(33656002)(5007970100001)(5002640100001)(50986999)(74316001)(5003600100002)(15975445007)(19580395003)(102836002)(2900100001)(66066001)(64706001)(10400500002)(76176999)(81156007)(122556002)(189998001)(105586002)(40100003)(92566002)(97736004)(2656002)(19580405001)(110136002)(4001540100001)(106356001)(5001860100001)(551544002)(99286002)(107886002)(5001830100001)(86362001)(5001960100002)(2351001);DIR:OUT;SFP:1101;SCL:1;SRVR:SN1PR0701MB1837;H:SN1PR0701MB1837.namprd07.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; received-spf: None (protection.outlook.com: redpoint.net does not designate permitted sender hosts) spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: multipart/alternative; boundary="_000_SN1PR0701MB18370F403C810D0BA6A78C5180670SN1PR0701MB1837_" MIME-Version: 1.0 X-OriginatorOrg: redpoint.net X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Aug 2015 22:47:39.9095 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 16a3d264-4987-408a-a6aa-69dd136253fc X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1PR0701MB1837 --_000_SN1PR0701MB18370F403C810D0BA6A78C5180670SN1PR0701MB1837_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable I don't see how to do this in our architecture. Our clients may be across V= PNs and completely inaccessible to the KDC. Our server basically functions = as the Hadoop client, even though it is a long-running service on the OS ho= st. When a user logs in, it performs these steps: - Spawn a new process using CreateProcessWithLogon on Windows, or fork/PAM/= setuid on Linux. - Use UsergroupInformation to login the user using the login/password and o= btain a ticket for the UGI's default user context. All of this works just fine, although we are forced to use reflection to ga= in access to the private setLogin() method of UGI. The thing is, we cannot simply login again in the same process to avoid tic= ket expiration. It doesn't work. I see that there is dedicated code in relo= ginFromKeytab() to clear out the old tokens and install new ones while keep= ing the UGI.class locked. We need to be able to do something similar given = the login and password. It would be great if there were an in-process API that completely mimicked = kinit and managed the TGT cache instead of the in-memory data of UGI. But w= e haven't found one. And we can't just run kinit, because it has no method = to inject a password (assume intentionally, since there isn't a way to secu= re that as a command-line argument). As I mentioned before, we can get tickets using login/password via UGI. Thi= s works. But I get lost around "how does one re-login and use the new ticke= t?". Here's what we observe: - We do a login and see the TGT in the UGI - After accessing HDFS, we see the TGT and the HDFS ticket in UGI. - After user.logout(), both tickets disappear - After a new login, a new TGT appears - Accessing HDFS fails and no HDFS ticket is added. We are trying to understand why this process diverges from reloginFromKeyta= b(), which works just fine. John Lilley From: Zheng, Kai [mailto:kai.zheng@intel.com] Sent: Monday, August 17, 2015 5:40 PM To: user@hadoop.apache.org Subject: RE: UserGroupInformation and login with password Hi John, Login from keytab is mostly expected for services. For end users, yes they = use passwords. In Kerberos (and Hadoop), it's expected for end users to exe= cute kinit like tool and generate ticket caches, then some methods like log= in from ticket cache in UGI will do the left work and help in your case. Or do you have to use the password directly in your program? If so, you may= add the method by yourself: 1) let your program prompt to user for passwor= d; 2) if your program has gathered the password in other means, then use so= me support like below: In Krb5LoginModule: * useFirstPass if, true, this LoginModule retrieves the * username and password from the module's shared state, * using "javax.security.auth.login.name" and * "javax.security.auth.login.password" as the respective * keys. The retrieved values are used for authentication. * If authentication fails, no attempt for a retry * is made, and the failure is reported back to the * calling application. Hope this helps. Regards, Kai From: John Lilley [mailto:john.lilley@redpoint.net] Sent: Monday, August 17, 2015 11:28 PM To: 'user@hadoop.apache.org' Subject: UserGroupInformation and login with password Greetings, Our software uses UserGroupInformation to authenticate with Kerberos-secure= clusters. We've found that there are obvious methods for logging in via k= eytab: loginUserFromKeytab() reloginFromKeytab() However, there are not obvious analogous methods for password-based login. = We've created the equivalent to loginUserFromPassword() using reflection t= o access private members, but have not yet created the equivalent reloginFr= omPassword(). It doesn't seem right to be using reflection here, but we cannot find the p= ublic API for principal/password login and relogin. It seems like this sho= uld be something simple. We do need to support password, because many of o= ur customers do not allow keytabs. Thanks John Lilley --_000_SN1PR0701MB18370F403C810D0BA6A78C5180670SN1PR0701MB1837_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

I don't see how to do = this in our architecture. Our clients may be across VPNs and completely ina= ccessible to the KDC. Our server basically functions as the Hadoop client, = even though it is a long-running service on the OS host. When a user logs in, it performs these steps:

– Spawn a new pr= ocess using CreateProcessWithLogon on Windows, or fork/PAM/setuid on Linux.=

– Use UsergroupI= nformation to login the user using the login/password and obtain a ticket f= or the UGI's default user context.

All of this works just= fine, although we are forced to use reflection to gain access to the priva= te setLogin() method of UGI.

The thing is, we canno= t simply login again in the same process to avoid ticket expiration. It doe= sn't work. I see that there is dedicated code in reloginFromKeytab() to cle= ar out the old tokens and install new ones while keeping the UGI.class locked. We need to be able to do somethin= g similar given the login and password.

It would be great if t= here were an in-process API that completely mimicked kinit and managed the = TGT cache instead of the in-memory data of UGI. But we haven't found one. A= nd we can't just run kinit, because it has no method to inject a password (assume intentionally, since there i= sn't a way to secure that as a command-line argument).

 

As I mentioned before,= we can get tickets using login/password via UGI. This works. But I get los= t around "how does one re-login and use the new ticket?". Here's = what we observe:

– We do a login = and see the TGT in the UGI

– After accessin= g HDFS, we see the TGT and the HDFS ticket in UGI.

– After user.log= out(), both tickets disappear

– After a new lo= gin, a new TGT appears

– Accessing HDFS= fails and no HDFS ticket is added.

We are trying to under= stand why this process diverges from reloginFromKeytab(), which works just = fine.

 

John Lilley

 

From: Zheng, Kai [mailto:kai.zheng@intel.com]=
Sent: Monday, August 17, 2015 5:40 PM
To: user@hadoop.apache.org
Subject: RE: UserGroupInformation and login with password=

 

Hi John,

 

Login from keytab is m= ostly expected for services. For end users, yes they use passwords. In Kerb= eros (and Hadoop), it’s expected for end users to execute kinit like = tool and generate ticket caches, then some methods like login from ticket cache in UGI will do the left work and help= in your case.

 

Or do you have to use = the password directly in your program? If so, you may add the method by you= rself: 1) let your program prompt to user for password; 2) if your program = has gathered the password in other means, then use some support like below:

In Krb5LoginModule:

*    us= eFirstPass   if, true, this LoginModule retrieves the<= /span>

*   &nb= sp;            =    username and password from the module's shared state,

*   &nb= sp;            =    using "javax.security.auth.login.name" and

*   &nb= sp;            =    "javax.security.auth.login.password" as the respecti= ve

*   &nb= sp;            =    keys. The retrieved values are used for authentication.

*   &nb= sp;            =    If authentication fails, no attempt for a retry

*   &nb= sp;            =    is made, and the failure is reported back to the

*   &nb= sp;            =    calling application.

 

Hope this helps.<= /o:p>

 

Regards,

Kai<= /p>

 

From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Monday, August 17, 2015 11:28 PM
To: 'user@hadoop.apache.org'
Subject: UserGroupInformation and login with password

 

Greetings,

 

Our software uses UserGroupInformation to authentica= te with Kerberos-secure clusters.  We’ve found that there are ob= vious methods for logging in via keytab:

loginUserFromKeytab()

reloginFromKeytab()

 

However, there are not obvious analogous methods for= password-based login.  We’ve created the equivalent to loginUse= rFromPassword() using reflection to access private members, but have not ye= t created the equivalent reloginFromPassword().

 

It doesn’t seem right to be using reflection h= ere, but we cannot find the public API for principal/password login and rel= ogin.  It seems like this should be something simple.  We do need= to support password, because many of our customers do not allow keytabs.

 

Thanks

John Lilley

 

--_000_SN1PR0701MB18370F403C810D0BA6A78C5180670SN1PR0701MB1837_--