hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yongzhi Wang <wang.yongzhi2...@gmail.com>
Subject Re: Hadoop Security and Kerberos
Date Tue, 18 Sep 2012 03:01:37 GMT
Thanks, Rekha,

This information is useful for me.

I have another question. Since I am using Debian 32-bit Linux, I need
the 32-bit binary file taskcontroller. However, I found the binary
files provided in hadoop 1.0.3 is 64 bit. I downloaded the hadoop
build file from server jenkins
(https://builds.apache.org/job/Hadoop-1.0-Build/ws/trunk/build/c++-build/Linux-i386-32/task-controller/).
It's still a 64 bit file.

I got the following errors when I start task tracker using the hadoop
64-bit taskcontroller:

12/09/17 11:59:58 ERROR mapred.TaskTracker: Can not start task tracker
because java.io.IOException: Task controller setup failed because of
invalidpermissions/ownership with exit code 126
        at org.apache.hadoop.mapred.LinuxTaskController.setup(LinuxTaskController.java:143)
        at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1452)
        at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3742)
Caused by: org.apache.hadoop.util.Shell$ExitCodeException:
/opt/ywang/hadoop-1.0.3/libexec/../bin/task-controller:
/opt/ywang/hadoop-1.0.3/libexec/../bin/task-controller: cannot execute
binary file

        at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
        at org.apache.hadoop.util.Shell.run(Shell.java:182)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
        at org.apache.hadoop.mapred.LinuxTaskController.setup(LinuxTaskController.java:137)

I am wondering if not providing 32-bit of taskcontroller is a build
bug, or 64-bit taskcontroller can be used somehow on the 32-bit
platform? If no 32-bit executable is provided in the daily build of
hadoop, how can I build one by myself?

Thanks!
Yongzhi

On Mon, Sep 17, 2012 at 5:42 AM, Joshi, Rekha <Rekha_Joshi@intuit.com> wrote:
> Hi Yongzhi,
>
> Well, I don't know if this will help, but I looked into source code, can
> see all token, authentication related features discussed in the design
> under- o.a.h.hdfs.security.*, o.a.h.mapreduce.security.*, o.a.h.security.*
> ,  o.a.h.security.authentication.*
> And HADOOP-4487 is marked fixed now, so there might be explicit bug issue,
> but features are in.
> Comparing the release notes can also give more details -
> http://hadoop.apache.org/docs/r1.0.3/releasenotes.html with
> http://hadoop.apache.org/docs/r1.0.0/releasenotes.html
>
> Owen session on security is good, albeit a bit old -
> http://developer.yahoo.com/blogs/ydn/posts/2010/07/hadoop_security_in_detai
> l/
> For kerberos itself, this is neat -
> http://www.ornl.gov/~jar/HowToKerb.html and
> http://www.cmf.nrl.navy.mil/krb/kerberos-faq.html
>
> So installing kerberos itself would be almost similar steps across CDH4,
> Hortonworks , Yahoo! - only configuration would need to be correctly setup
> in kerberos.principal, authentication.type in core-site.xml
> Some more examples -
> http://hortonworks.com/blog/fine-tune-your-apache-hadoop-security-settings/
> #more-1124
> https://cwiki.apache.org/GIRAPH/quick-start-running-giraph-with-secure-hado
> op.html
>
> Thanks
> Rekha
>
>
>
> On 16/09/12 8:57 AM, "Yongzhi Wang" <wang.yongzhi2009@gmail.com> wrote:
>
>>Dear All,
>>
>>I am confused about the usage of Kerberos on Hadoop 1.0.3.
>>
>>I have difficulty in finding some documents to configure of the
>>security feature of HADOOP 1.0.3. Specifically, how should I configure
>>the Hadoop, so that I can use Kerberos? The only document that is
>>related with this question is CDH4 Security Guide
>>(https://ccp.cloudera.com/display/CDH4DOC/CDH4+Security+Guide), an
>>instruction about the security configuration for CloudEra Distributed
>>Hadoop. But I am not sure if this guide can be directly used to
>>configure the Apache Hadoop 1.0.3. Afterall, I don't know how many
>>differences exist between the CDH4 and Apache Hadoop 1.0.3.
>>
>>I read some materials published by the hadoop development team,
>>including the documentation posted on the apache website
>>(http://hadoop.apache.org/docs/r1.0.3/index.html) and the "Hadoop
>>Security Design" document proposed by Yahoo! in 2009. Unfortunately, I
>>still can not generate a clear vision after I read those documents.
>>All my questions are derived from one basic question: Are all of the
>>design features in "Hadoop Security Design" included in the release
>>1.0.3? If not, which of those features are introduced in release
>>1.0.3? Which features are included in the Hadoop 2.0? Which features
>>are still not implemented?
>>
>>For example, the "Hadoop Security Design" document mentioned three
>>types of tokens (Delegation Token, Block Access Token and Job Token).
>>Did release 1.0.3 support all the three types of tokens?
>>
>>In the 1.0.3 document "hdfs permission guide"
>>(http://hadoop.apache.org/docs/r1.0.3/hdfs_permissions_guide.html), it
>>mentions that "In this release of Hadoop the identity of a client
>>process is just whatever the host operating system says it is. For
>>Unix-like systems, ......In the future there will be other ways of
>>establishing user identity (think Kerberos, LDAP, and others).
>>......". It seems the 1.0.3 does not fully support Kerberos. If in
>>that case, to what degree does the release 1.0.3 support Kerberos?
>>
>>So my question is:
>>
>> 1. Is there any document comparing the security feature in each
>>release of hadoop with the "Hadoop Security Design" proposed by Yahoo!
>>?
>> 2. In release 1.0.3, which component of hadoop can use Kerberos to
>>leverage security? In order to use the Kerberos, how should I
>>configure Hadoop?
>>
>>I am not very familiar with Kerberos. So if I have some
>>misunderstanding, please feel free to point out.
>>
>>Thanks!
>>
>>Best regards,
>>Yongzhi
>

Mime
View raw message