Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 40BD5115B0 for ; Wed, 27 Aug 2014 01:00:52 +0000 (UTC) Received: (qmail 52584 invoked by uid 500); 27 Aug 2014 01:00:47 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 52431 invoked by uid 500); 27 Aug 2014 01:00:47 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 52421 invoked by uid 99); 27 Aug 2014 01:00:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Aug 2014 01:00:47 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of john.lilley@redpoint.net designates 207.46.163.144 as permitted sender) Received: from [207.46.163.144] (HELO na01-bn1-obe.outbound.protection.outlook.com) (207.46.163.144) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Aug 2014 01:00:40 +0000 Received: from DM2PR0701MB729.namprd07.prod.outlook.com (10.242.126.152) by DM2PR0701MB730.namprd07.prod.outlook.com (10.242.126.153) with Microsoft SMTP Server (TLS) id 15.0.1015.19; Wed, 27 Aug 2014 01:00:17 +0000 Received: from DM2PR0701MB729.namprd07.prod.outlook.com ([10.242.126.152]) by DM2PR0701MB729.namprd07.prod.outlook.com ([10.242.126.152]) with mapi id 15.00.1015.018; Wed, 27 Aug 2014 01:00:17 +0000 From: John Lilley To: "user@hadoop.apache.org" Subject: RE: winutils and security Thread-Topic: winutils and security Thread-Index: Ac+/ERE/gT3oWddLTIyPO4WzqKfmKQArE1XgAHUVJjA= Date: Wed, 27 Aug 2014 01:00:16 +0000 Message-ID: <7a3a551814924508805ae4954bbe87b7@DM2PR0701MB729.namprd07.prod.outlook.com> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [98.245.83.119] x-microsoft-antispam: BCL:0;PCL:0;RULEID:;UriScan:; x-forefront-prvs: 0316567485 x-forefront-antispam-report: SFV:NSPM;SFS:(199003)(189002)(377454003)(377424004)(19580405001)(21056001)(83322001)(19580395003)(76576001)(19300405004)(99396002)(19625215002)(76176999)(101416001)(74316001)(16236675004)(81342001)(83072002)(15975445006)(87936001)(15202345003)(85852003)(86362001)(33646002)(92566001)(106356001)(105586002)(74662001)(64706001)(20776003)(99286002)(74502001)(108616004)(80022001)(95666004)(31966008)(76482001)(66066001)(77982001)(90102001)(54356999)(50986999)(81542001)(2351001)(85306004)(4396001)(110136001)(551544002)(107046002)(2656002)(79102001)(46102001)(107886001)(2501001)(24736002)(460985004);DIR:OUT;SFP:;SCL:1;SRVR:DM2PR0701MB730;H:DM2PR0701MB729.namprd07.prod.outlook.com;FPR:;MLV:sfv;PTR:InfoNoRecords;A:1;MX:1;LANG:en; Content-Type: multipart/alternative; boundary="_000_7a3a551814924508805ae4954bbe87b7DM2PR0701MB729namprd07p_" MIME-Version: 1.0 X-OriginatorOrg: redpoint.net X-Virus-Checked: Checked by ClamAV on apache.org --_000_7a3a551814924508805ae4954bbe87b7DM2PR0701MB729namprd07p_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable One more follow up, in case someone stumbles across this in the future. Fr= om what we can tell, the Hadoop security initialization is very sensitive t= o startup order, and this has been confirmed by discussions with other peop= le. The only thing that we've been able to make work at all reliably uses = the following sequence, in a single thread, preferably very close to startu= p. 1. Load/set Configuration that can be used by HDFS and YARN. 2. Set UserGroupInformation() and log in using either password or key= tab. 3. Open the HDFS FileSystem 4. Call addDelegationTokens() to extract delegated Credentials for HD= FS and keep them around. Once this has been done, it appears tha tall is well. We can use those Cre= dentials in the YARN application master launch context. john From: John Lilley [mailto:john.lilley@redpoint.net] Sent: Sunday, August 24, 2014 11:05 AM To: user@hadoop.apache.org Subject: RE: winutils and security Following up on this, I was able to extract a winutils.exe and Hadoop.dll f= rom a Hadoop install for Windows, and set up HADDOP_HOME and PATH to find t= hem. It makes no difference to security, apparently. John From: John Lilley [mailto:john.lilley@redpoint.net] Sent: Saturday, August 23, 2014 2:41 PM To: 'user@hadoop.apache.org' Subject: winutils and security Up to this point, we've been able to run as a Hadoop client application (HD= FS + YARN) from Windows without winutils.exe, despite always seeing message= s complaining about it in the logs. However, we are now integrating with s= ecure clusters and are having some mysterious errors. Before these errors = occur, messages come from Hadoop like those below. Is it possible that thi= s is leading to our security failures? (I posted previously about that prob= lem but got no response). What does winutils.exe have to do with security,= if anything? Thanks john The relevant portions of the log seem to be: 2014-08-23 14:33:10 DEBUG org.apache.hadoop.metrics2.impl.MetricsSystemImpl= : UgiMetrics, User and group related metrics 2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.Groups: Creating new = Groups object 2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: Trying t= o load the custom-built native-hadoop library... 2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: Failed t= o load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop = in java.library.path 2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: java.lib= rary.path=3D[...] 2014-08-23 14:33:10 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to= load native-hadoop library for your platform... using builtin-java classes where applicable 2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.JniBasedUnixGroupsMapp= ingWithFallback: Falling back to shell based 2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.JniBasedUnixGroupsMapp= ingWithFallback: Group mapping impl=3Dorg.apache.hadoop.security.ShellBased= UnixGroupsMapping 2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.Groups: Group mapping = impl=3Dorg.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; ca= cheTimeout=3D300000 --_000_7a3a551814924508805ae4954bbe87b7DM2PR0701MB729namprd07p_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

One more follow up, in= case someone stumbles across this in the future.  From what we can te= ll, the Hadoop security initialization is very sensitive to startup order, = and this has been confirmed by discussions with other people.  The only thing that we’ve been able to make= work at all reliably uses the following sequence, in a single thread, pref= erably very close to startup.

1.&n= bsp;      Load/set Confi= guration that can be used by HDFS and YARN.

2.&n= bsp;      Set UserGroupI= nformation() and log in using either password or keytab.<= /p>

3.&n= bsp;      Open the HDFS = FileSystem

4.&n= bsp;      Call addDelega= tionTokens() to extract delegated Credentials for HDFS and keep them around= .

 

Once this has been don= e, it appears tha tall is well.  We can use those Credentials in the Y= ARN application master launch context.

 

john=

 

From: John Lilley [mailto:john.lilley@redpoin= t.net]
Sent: Sunday, August 24, 2014 11:05 AM
To: user@hadoop.apache.org
Subject: RE: winutils and security

 

Following up on this, = I was able to extract a winutils.exe and Hadoop.dll from a Hadoop install f= or Windows, and set up HADDOP_HOME and PATH to find them.  It makes no= difference to security, apparently.

 

John=

 

From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Saturday, August 23, 2014 2:41 PM
To: 'user@hadoop.apache.org'
Subject: winutils and security

 

Up to this point, we’ve been able to run as a = Hadoop client application (HDFS + YARN) from Windows without winutils.e= xe, despite always seeing messages complaining about it in the logs.  = However, we are now integrating with secure clusters and are having some mysterious errors.  Before these errors occur, me= ssages come from Hadoop like those below.  Is it possible that this is= leading to our security failures? (I posted previously about that problem = but got no response).  What does winutils.exe have to do with security, if anything?

 

Thanks

john

 

The relevant portions of the log seem to be:

 

2014-08-23 14:33:10 DEBUG org.apache.hadoop.metrics2= .impl.MetricsSystemImpl: UgiMetrics, User and group related metrics

2014-08-23 14:33:10 DEBUG org.apache.hadoop.security= .Groups:  Creating new Groups object

2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.Nat= iveCodeLoader: Trying to load the custom-built native-hadoop library...

2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.Nat= iveCodeLoader: Failed to load native-hadoop with error: java.lang.Unsatisfi= edLinkError: no hadoop in java.library.path

2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.Nat= iveCodeLoader: java.library.path=3D[…]

2014-08-23 14:33:10 WARN org.apache.hadoop.util.Nati= veCodeLoader: Unable to load native-hadoop library for your platform... usi= ng

builtin-java classes where applicable

2014-08-23 14:33:10 DEBUG org.apache.hadoop.security= .JniBasedUnixGroupsMappingWithFallback: Falling back to shell based

2014-08-23 14:33:10 DEBUG org.apache.hadoop.security= .JniBasedUnixGroupsMappingWithFallback: Group mapping impl=3Dorg.apache.had= oop.security.ShellBasedUnixGroupsMapping

2014-08-23 14:33:10 DEBUG org.apache.hadoop.security= .Groups: Group mapping impl=3Dorg.apache.hadoop.security.JniBasedUnixGroups= MappingWithFallback; cacheTimeout=3D300000

--_000_7a3a551814924508805ae4954bbe87b7DM2PR0701MB729namprd07p_--