hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From André Kelpe (JIRA) <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9923) yarn staging area on hdfs has wrong permission and is created by the wrong user
Date Mon, 02 Sep 2013 10:28:52 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13756006#comment-13756006

André Kelpe commented on HADOOP-9923:

bigtop for sure can help, but shouldn't the hadoop distribution itself be deployable "as-is",
even if it is not the most optimal way? 

The fact, that the history server creates a directory, that it is only supposed to read from,
is IMO a bug. If it is not a bug, then it should at least be documented, how to initialize
hdfs correctly, by just using the hadoop tarball. It is somewhat mentioned in the docs, but
the fact that you have to do that in the middle of the daemon start-ups ins non-obvious. At
least to me. 

It is also a regression from hadoop 1.x, where you could fire up a cluster with start-all.sh
and it worked. (I know that start-all.sh is not the most ideal way, but it does work, for
certain use cases.)
> yarn staging area on hdfs has wrong permission and is created by the wrong user
> -------------------------------------------------------------------------------
>                 Key: HADOOP-9923
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9923
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 2.1.0-beta
>            Reporter: André Kelpe
> I am setting up a cluster with hadoop 2.1-beta that consists of the following components:
> master: runs the namenode, the resourcemanager and the job history server.
> hadoop1, hadoop2, hadoop3: run datanodes and node managers
> I created 3 system users for the different components, like explained in the docs:
> hdfs: runs all things hdfs
> yarn: runs all things yarn
> mapred: runs the job history server
> If I now boot up the cluster, I cannot submit jobs since the yarn staging area permissions
do not allow it.
> What I found out is, that the job-history-server is creating the staging directory while
starting. This first of all causes it to be owned by the wrong user (mapred) and having the
wrong permision (770). The docs are not really clear if I am supposed to start hdfs first,
then create the staging area by hand and then start the job-history-server or if this is supposed
to happen automatically by itself.
> In any case, either the docs should be updated or the job-history-server should not create
the directory.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message