Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4EF02105B5 for ; Wed, 11 Dec 2013 02:42:28 +0000 (UTC) Received: (qmail 61801 invoked by uid 500); 11 Dec 2013 02:42:23 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 61700 invoked by uid 500); 11 Dec 2013 02:42:23 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 61693 invoked by uid 99); 11 Dec 2013 02:42:22 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Dec 2013 02:42:22 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jayunit100@gmail.com designates 209.85.214.182 as permitted sender) Received: from [209.85.214.182] (HELO mail-ob0-f182.google.com) (209.85.214.182) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Dec 2013 02:42:19 +0000 Received: by mail-ob0-f182.google.com with SMTP id wp4so6300270obc.13 for ; Tue, 10 Dec 2013 18:41:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=b8quG+qIH1VAFW0AsF///3gkBAkcO9Nwz/Z4kAWDvvg=; b=npIoo0kwi5QRzszx8RRGl6pajEFe2nZ2WqmBTou9YpQ5n8S/GeOgyzCuc0EutJySot HBnxktpLGRjEwO3Ya9SHDz9hzHbawrjj/RKVShxoSnic5I9jI4O40/rW+qIHhStzayXT /Qbl+Xnlime9dWMMrJNP9vgHTNjbNTKrCabsEX8aNJh2QX0emgfm1u1nE+PBNWAtIpRZ A2jJ2/RjhKyTwuPcUvRmwgGbT7n+5/cvesnXAaivdKODtWXnDfNFi2U0o/a8F8GUUzTf qa8EQyweVeH+qtMt5gJCUi9dqF0gk7OOf8mwZRtabfaTt9QDdm5pOhivHNW8BNC+F+B6 kXKw== MIME-Version: 1.0 X-Received: by 10.60.174.167 with SMTP id bt7mr4096793oec.54.1386729718207; Tue, 10 Dec 2013 18:41:58 -0800 (PST) Received: by 10.76.74.70 with HTTP; Tue, 10 Dec 2013 18:41:58 -0800 (PST) In-Reply-To: References: Date: Tue, 10 Dec 2013 21:41:58 -0500 Message-ID: Subject: Re: multiusers in hadoop through LDAP From: Jay Vyas To: "common-user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=047d7bd6c03e40db8e04ed392c98 X-Virus-Checked: Checked by ClamAV on apache.org --047d7bd6c03e40db8e04ed392c98 Content-Type: text/plain; charset=ISO-8859-1 So, not knowing much about LDAP, but being very interested in the multiuser problem on multiuser filesystems, i was excited to see this question.... Im researching the same thing at the moment, and it seems obviated by the fact that : - the FileSystem API itslef provides implementations for getting group and user names / permissions.... And furthermore - the linux task controllers launch jobs as the user submitting the job, whereas the regular task controllers launch tasksunder the YARN daemon name, iirc. So.... where does LDAP begin and TaskController / FileSystem notions of ownership end.... ? I guess I'm also asking what are the entites which are "ownable" in hadoop app , and how we can leverage the GroupMappingServiceProviders to deploy more flexible hadoop environments. Any thoughts on this would be appreciated. On Tue, Dec 10, 2013 at 6:38 PM, Adam Kawa wrote: > Please have a look at hadoop.security.group.mapping.ldap.* settings as Hardik > Pandya suggests. > > ===== > > In advance, just to share our story related to LDAP + > hadoop.security.group.mapping.ldap.*, if you run into the same limitation > as we did: > > In many cases hadoop.security.group.mapping.ldap.* should solve your > problem. Unfortunately, they did now work for us. The problematic setting > relates to an additional filter to use when searching for LDAP groups. We > wanted to use posixGroups filter, but it is currently not supported by > Hadoop. Finally, we found a workaround using name service switch > configuration where we specified that the LDAP should the primary source of > information about groups of our users. This means that we solved this > problem on the operating system level, not on Hadoop level. > > You can read more about this issue here: > > http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/ > and here > http://www.slideshare.net/AdamKawa/hadoop-adventures-at-spotify-strata-conference-hadoop-world-2013 (slides > 18-26). > > > 2013/12/10 Hardik Pandya > >> >> have you looked at hadoop.security.group.mapping.ldap.* in >> hadoop-common/core-default.xml >> >> additional resourcemay help >> >> >> >> >> >> >> On Tue, Dec 10, 2013 at 3:06 AM, YouPeng Yang wrote: >> >>> Hi >>> >>> In my cluster ,I want to have multiusers for different purpose.The >>> usual method is to add a user through the OS on Hadoop NameNode . >>> I notice the hadoop also support to LDAP, could I add user through >>> LDAP instead through OS? So that if a user is authenticated by the LDAP >>> ,who will also access the HDFS directory? >>> >>> >>> Regards >>> >> >> > -- Jay Vyas http://jayunit100.blogspot.com --047d7bd6c03e40db8e04ed392c98 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
So, not knowing much about LDAP, but being very inter= ested in the multiuser problem on multiuser filesystems, i was excited to s= ee this question.... Im researching the same thing at the moment, and it se= ems obviated by the fact that :

- the FileSystem API itslef provides implementations for getting group = and user names / permissions....

And furthermore

- the linux task controllers launch jobs as th= e user submitting the job, whereas the regular task controllers launch task= sunder the YARN daemon name, iirc.

So.... where does LDAP begin and TaskC= ontroller / FileSystem notions of ownership end.... ?=A0

I guess I'm also asking what are the entites whi= ch are "ownable" in hadoop app , and how we can leverage the Grou= pMappingServiceProviders to deploy more flexible hadoop environments.

Any thoughts on this would be appreciated.

On Tue, Dec 10, 2013 at 6:38 PM, Ad= am Kawa <kawa.adam@gmail.com> wrote:
Please have a look at=A0hadoop.security.gr= oup.mapping.ldap.* settings as=A0Hardik Pan= dya suggests.

=3D=3D=3D=3D=3D

In=A0advance, ju= st to share our story related to LDAP +=A0hadoop.security.group.mapping.ldap.*, if you run = into the same limitation as we did:

In many cases=A0hadoop.security.group.mapping.ldap.*=A0should solve your problem. Unfortunately, they did now work for us. The pro= blematic setting relates to an additional filter to use when searching for = LDAP groups. We wanted to use posixGroups filter, but it is currently not s= upported by Hadoop. Finally, we found a workaround using name service switc= h configuration where we specified that the LDAP should the primary source = of information about groups of our users. This means that we solved this pr= oblem on the operating system level, not on Hadoop level.


2013/12/10 Hardik Pandya <smarty.juice@gmail.com>

have you looked at hadoop.security.group.mapping.= ldap.* in=A0 hadoop-common/cor= e-default.xml

additional resource may help






On Tue, Dec 10, 2013 at 3:06 AM, YouPeng Yan= g <yypvsxf19870706@gmail.com> wrote:
Hi
=A0
=A0 In my cluster ,I wa= nt to have multiusers for different purpose.The usual method is to add a us= er through the OS=A0 on=A0 Hadoop NameNode .
=A0 I notice the hado= op also support to LDAP, could I add user through LDAP instead through OS? = So that if a user is authenticated by the LDAP ,who will also access the HD= FS directory?


Regards





--
Jay Vyashttp://jayuni= t100.blogspot.com
--047d7bd6c03e40db8e04ed392c98--