hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haibo Chen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-5001) Aggregated Logs root directory is created with wrong owner and group if nonexistent
Date Tue, 26 Apr 2016 18:12:13 GMT

     [ https://issues.apache.org/jira/browse/YARN-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Haibo Chen updated YARN-5001:
-----------------------------
    Description: 
Usually, the group owner for /tmp/logs, where the aggregated logs go, is "hadoop". Under that
dir, you then have <username>/logs/<application_id>/<files_per_node> with
group being "hadoop" all the way down. 

If you delete the /tmp/logs dir (when you want to clean up all the logs), the directory will
be created with a different group "superuser". The JHS runs as the mapred user, who is a member
of the hadoop group. With the new group, the JHS doesn't have permission to read the logs
any more.

  was:
Usually, the owner and group for /tmp/logs, where the aggregated logs go, is:  
[root@nightly57-1 ~]# hadoop fs -ls /tmp/ | grep logs
drwxrwxrwt   - mapred    hadoop              0 2016-04-14 15:46 /tmp/logs
Under that dir, you then have <username>/logs/<application_id>/<files_per_node>.
The group should be hadoop all the way down, while the user should be mapred at the top, and
<username> starting with the <username> dir and below.
If you delete the /tmp/logs dir (when you want to clean up all the logs):
[root@nightly57-1 ~]# sudo -u hdfs hadoop fs -rmr /tmp/logs
And then run an MR job:
[root@nightly57-1 ~]# hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi
2 2
…
The directory will be created with a different owner and group:
[root@nightly57-1 ~]# hadoop fs -ls /tmp/ | grep logs
drwxrwxrwt   - yarn      supergroup          0 2016-04-14 18:12 /tmp/logs

The owner being yarn might be okay, though this is inconsistent with the original owner, mapred.
However, the real problem is the group now being supergroup instead of hadoop. The JHS runs
as the mapred user, who is a member of the hadoop group. With the new owner and group, the
JHS doesn't have permission to read the logs any more.


> Aggregated Logs root directory is created with wrong owner and group if nonexistent 
> ------------------------------------------------------------------------------------
>
>                 Key: YARN-5001
>                 URL: https://issues.apache.org/jira/browse/YARN-5001
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.7.0
>            Reporter: Haibo Chen
>            Assignee: Haibo Chen
>
> Usually, the group owner for /tmp/logs, where the aggregated logs go, is "hadoop". Under
that dir, you then have <username>/logs/<application_id>/<files_per_node>
with group being "hadoop" all the way down. 
> If you delete the /tmp/logs dir (when you want to clean up all the logs), the directory
will be created with a different group "superuser". The JHS runs as the mapred user, who is
a member of the hadoop group. With the new group, the JHS doesn't have permission to read
the logs any more.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message