hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haibo Chen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-5001) Aggregated Logs root directory is created with wrong owner and group if nonexistent
Date Tue, 26 Apr 2016 18:08:12 GMT
Haibo Chen created YARN-5001:
--------------------------------

             Summary: Aggregated Logs root directory is created with wrong owner and group
if nonexistent 
                 Key: YARN-5001
                 URL: https://issues.apache.org/jira/browse/YARN-5001
             Project: Hadoop YARN
          Issue Type: Bug
          Components: nodemanager
    Affects Versions: 2.7.0
            Reporter: Haibo Chen
            Assignee: Haibo Chen


Usually, the owner and group for /tmp/logs, where the aggregated logs go, is:  
[root@nightly57-1 ~]# hadoop fs -ls /tmp/ | grep logs
drwxrwxrwt   - mapred    hadoop              0 2016-04-14 15:46 /tmp/logs
Under that dir, you then have <username>/logs/<application_id>/<files_per_node>.
The group should be hadoop all the way down, while the user should be mapred at the top, and
<username> starting with the <username> dir and below.
If you delete the /tmp/logs dir (when you want to clean up all the logs):
[root@nightly57-1 ~]# sudo -u hdfs hadoop fs -rmr /tmp/logs
And then run an MR job:
[root@nightly57-1 ~]# hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi
2 2
…
The directory will be created with a different owner and group:
[root@nightly57-1 ~]# hadoop fs -ls /tmp/ | grep logs
drwxrwxrwt   - yarn      supergroup          0 2016-04-14 18:12 /tmp/logs

The owner being yarn might be okay, though this is inconsistent with the original owner, mapred.
However, the real problem is the group now being supergroup instead of hadoop. The JHS runs
as the mapred user, who is a member of the hadoop group. With the new owner and group, the
JHS doesn't have permission to read the logs any more.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message