hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stuti Awasthi <stutiawas...@hcl.com>
Subject RE: Security in Hadoop-1.0.0
Date Wed, 15 Feb 2012 08:47:34 GMT
What are the thoughts on hdfsproxy ? 
http://hadoop.apache.org/hdfs/docs/r0.21.0/hdfsproxy.html 

Will this be useful in my scenario ?

-----Original Message-----
From: Stuti Awasthi 
Sent: Wednesday, February 15, 2012 12:28 PM
To: hdfs-user@hadoop.apache.org
Subject: RE: Security in Hadoop-1.0.0

Thanks Harsh,

My usecase is : there will be multiple users which will connect to HDFS cluster using an application
for all filesystem operation. Normally if user login to app, their permissions on HDFS cluster
will be handle by the application front.
For any user who wants to connect to hdfs cluster directly without application, I want to
provide a security layer using LDAP in between so that only authenticated users can access
the cluster.

Since there will be n numbers of users, I do not want to add them to unix groups. I hope this
clears out my scenario.

Thanks for providing links and explaining them in detail. I will read more and try to implement
this at my end.

Stuti Awasthi


-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Tuesday, February 14, 2012 7:14 PM
To: hdfs-user@hadoop.apache.org
Subject: Re: Security in Hadoop-1.0.0

Stuti,

What are you looking for, exactly? Are all you asking for is strong authentication for your
HDFS clusters such that no external user may connect to it and read files (even those marked
o+r)? If so, that is what a HDFS security configuration, which we have pointed you to already,
aims to provide.

Know that LDAP isn't an "authentication" mechanism - and thats not what you want to "integrate"
HDFS with, for security. You need a functional Kerberos environment that integrates with your
LDAP, for strong authentication of users (token based security). To setup Kerberos integrated
with your existing LDAP service, please follow articles such as http://www.linux-mag.com/id/4738/

Once your Kerberos instance is setup to talk and authenticate users on your LDAP instance,
carry on with the guide pointed out earlier at
https://ccp.cloudera.com/display/CDHDOC/Configuring+Hadoop+Security+in+CDH3
- which will essentially work for Apache Hadoop 1.x too. You only need to bother with Kerberos
after this point.

Hope this clears it up for you.

P.s. If your environment already uses Active Directory to manage users, you can use that directly
as well:
https://ccp.cloudera.com/display/CDHDOC/Integrating+Hadoop+Security+with+Active+Directory

P.p.s. The doc page at
https://ccp.cloudera.com/display/CDHDOC/CDH3+Security+Guide carries further articles on Kerberos
and other security configs if you want to read more - and all of the instructions would work
with most upstream releases too.

On Tue, Feb 14, 2012 at 1:48 PM, Stuti Awasthi <stutiawasthi@hcl.com> wrote:
> After some googling I found the following link :
> http://mapredit.blogspot.in/2011/10/secure-your-hadoop-cluster-part-i.
> html
> http://mapredit.blogspot.in/2011/10/secure-your-hadoop-cluster-part-ii
> .html
>
> But these mainly deals with applying LDAP for map-reduce. I want to configure LDAP for
HDFS as well as mapreduce. Please suggest me some links through which I can configure dfs
with LDP also.
>
> Thanks
>
> -----Original Message-----
> From: Stuti Awasthi
> Sent: Tuesday, February 14, 2012 12:28 PM
> To: hdfs-user@hadoop.apache.org
> Subject: RE: Security in Hadoop-1.0.0
>
> Thanks Patrick,
>
> The concept is clear to me now. As a first step I would like to configure LDAP with Hadoop.
> I am using Apache Hadoop 1.0.0 but not able to find configuration steps in this version
documentation.
> It would be really helpful if someone can point me to relevant documentation of configuring
this version of Hadoop with LDAP.
>
> Thanks
>
> From: Patrick Angeles [mailto:patrickangeles@gmail.com]
> Sent: Monday, February 13, 2012 8:29 PM
> To: hdfs-user@hadoop.apache.org
> Subject: Re: Security in Hadoop-1.0.0
>
> LDAP and Kerberos are orthogonal in Hadoop, but both are often used together. LDAP allows
for centralized user/group management (sort of like DNS for your users). Kerberos is for strong
authentication of users.
>
> When using Kerberos in Hadoop, you want to propagate user/group identities to all your
cluster nodes. (Otherwise, you might authenticate strongly, but your user ID doesn't exist
in a Tasktracker so your job fails.) LDAP happens to be a common way to do this.
>
> Typically when you set up Kerberos, you also set up your cluster nodes to do LDAP authentication.
You do this setup at the operating system level (via PAM).
>
> Note that you can also use Hue as your user-gateway to Hadoop. In this scenario, you
can use an LDAP backend to authenticate users. You do not have to (but can) configure Hadoop
with Kerberos.
>
> - P
> On Mon, Feb 13, 2012 at 3:11 AM, Stuti Awasthi <stutiawasthi@hcl.com> wrote:
> Hi,
> I am bit confused on Security part of Hadoop. Cluster is behind the firewall. I have
read that Hadoop can be configured with LDAP also.
> I want to know which is better : configure Hadoop security with LDAP or Kerberos as both
provide authentication.
>
> Please provide me more details on this as I am newbee in this part.
>
> Thanks
>
>
> -----Original Message-----
> From: alo alt [mailto:wget.null@googlemail.com]
> Sent: Monday, February 06, 2012 3:56 PM
> To: hdfs-user@hadoop.apache.org
> Subject: Re: Security in Hadoop-1.0.0
>
> Kerberos tokens and lifetime:
> http://hortonworks.com/the-role-of-delegation-tokens-in-apache-hadoop-
> security/
>
> Security in CDH3 (the same as hadoop)
> https://ccp.cloudera.com/display/CDHDOC/CDH3+Security+Guide
>
> best,
>  Alex
>
> --
> Alexander Lorenz
> http://mapredit.blogspot.com
>
> On Feb 6, 2012, at 11:19 AM, Stuti Awasthi wrote:
>
>> Hi all,
>> I started looking into configure security in Hadoop-1.0.0 but do not find concrete
documentation on which kind of security is provided in this release and how to configure them.
>> Currently I am following
>> "http://hadoop.apache.org/common/docs/r1.0.0/" documentation
>>
>> As per knowledge, Proxy authentication and Kerberos security is provided in this
release of Hadoop. Please point me to some good documentation or give me some pointers from
where I can start this work.
>>
>> Thanks
>> Stuti Awasthi
>>
>>
>>
>> ::DISCLAIMER::
>> ---------------------------------------------------------------------
>> -
>> -------------------------------------------------
>>
>> The contents of this e-mail and any attachment(s) are confidential and intended for
the named recipient(s) only.
>> It shall not attach any liability on the originator or HCL or its 
>> affiliates. Any views or opinions presented in this email are solely those of the
author and may not necessarily reflect the opinions of HCL or its affiliates.
>> Any form of reproduction, dissemination, copying, disclosure, 
>> modification, distribution and / or publication of this message 
>> without the prior written consent of the author of this e-mail is 
>> strictly prohibited. If you have received this email in error please delete it and
notify the sender immediately. Before opening any mail and attachments please check them for
viruses and defect.
>>
>> ---------------------------------------------------------------------
>> -
>> -------------------------------------------------
>



--
Harsh J
Customer Ops. Engineer
Cloudera | http://tiny.cloudera.com/about

Mime
View raw message