hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From YouPeng Yang <yypvsxf19870...@gmail.com>
Subject Re: multiusers in hadoop through LDAP
Date Wed, 11 Dec 2013 08:21:14 GMT
Hi
 Thanks a lot for your replies.

 I will try the LDAP+hadoop.security.group.mapping.ldap.*. Right now I can
not catch this question.


Regards.


2013/12/11 Jay Vyas <jayunit100@gmail.com>

> So, not knowing much about LDAP, but being very interested in the
> multiuser problem on multiuser filesystems, i was excited to see this
> question.... Im researching the same thing at the moment, and it seems
> obviated by the fact that :
>
> - the FileSystem API itslef provides implementations for getting group and
> user names / permissions....
>
> And furthermore
>
> - the linux task controllers launch jobs as the user submitting the job,
> whereas the regular task controllers launch tasksunder the YARN daemon
> name, iirc.
>
> So.... where does LDAP begin and TaskController / FileSystem notions of
> ownership end.... ?
>
> I guess I'm also asking what are the entites which are "ownable" in hadoop
> app , and how we can leverage the GroupMappingServiceProviders to deploy
> more flexible hadoop environments.
>
> Any thoughts on this would be appreciated.
>
> On Tue, Dec 10, 2013 at 6:38 PM, Adam Kawa <kawa.adam@gmail.com> wrote:
>
>> Please have a look at hadoop.security.group.mapping.ldap.* settings as Hardik
>> Pandya suggests.
>>
>> =====
>>
>> In advance, just to share our story related to LDAP +
>> hadoop.security.group.mapping.ldap.*, if you run into the same
>> limitation as we did:
>>
>> In many cases hadoop.security.group.mapping.ldap.* should solve your
>> problem. Unfortunately, they did now work for us. The problematic
>> setting relates to an additional filter to use when searching for LDAP
>> groups. We wanted to use posixGroups filter, but it is currently not
>> supported by Hadoop. Finally, we found a workaround using name service
>> switch configuration where we specified that the LDAP should the primary
>> source of information about groups of our users. This means that we solved
>> this problem on the operating system level, not on Hadoop level.
>>
>> You can read more about this issue here:
>>
>> http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/
>> and here
>> http://www.slideshare.net/AdamKawa/hadoop-adventures-at-spotify-strata-conference-hadoop-world-2013
(slides
>> 18-26).
>>
>>
>> 2013/12/10 Hardik Pandya <smarty.juice@gmail.com>
>>
>>>
>>> have you looked at hadoop.security.group.mapping.ldap.* in
>>> hadoop-common/core-default.xml<http://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/core-default.xml>
>>>
>>> additional resource<http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/>may
help
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Dec 10, 2013 at 3:06 AM, YouPeng Yang <yypvsxf19870706@gmail.com
>>> > wrote:
>>>
>>>> Hi
>>>>
>>>>   In my cluster ,I want to have multiusers for different purpose.The
>>>> usual method is to add a user through the OS  on  Hadoop NameNode .
>>>>   I notice the hadoop also support to LDAP, could I add user through
>>>> LDAP instead through OS? So that if a user is authenticated by the LDAP
>>>> ,who will also access the HDFS directory?
>>>>
>>>>
>>>> Regards
>>>>
>>>
>>>
>>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>

Mime
View raw message