hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martin Gerlach (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-4557) With default settings, log aggregation service creates aggregated log dirs with group ownership not matching JH server run-as user
Date Wed, 15 Aug 2012 15:42:38 GMT
Martin Gerlach created MAPREDUCE-4557:
-----------------------------------------

             Summary: With default settings, log aggregation service creates aggregated log
dirs with group ownership not matching JH server run-as user
                 Key: MAPREDUCE-4557
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4557
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: nodemanager
    Affects Versions: 2.0.0-alpha
         Environment: CDH 4.0.1 (ZK, HDFS, YARN, HBase) managed by Cloudera Manager 4.0.3
            Reporter: Martin Gerlach
            Priority: Minor


In order to read aggregated logs, JH server, running as mapred:hadoop by default, tries to
access hdfs://<host:port>/tmp/logs/<user>/logs/<appId>/...

NodeManager runs as yarn:hadoop by default, but creates /tmp/logs initially as user yarn,
and group unchanged. E.g., if /tmp as ownership hdfs:supergroup, /tmp/logs will have ownership
yarn:supergroup.

Upon running a job, /tmp/logs/<username> is created by LogAggregationService as the
user who submitted the job and leaves the group unchanged, e.g., /tmp/logs/<user> will
have ownership <user>:supergroup, and permissions 750.

Like this, JH server, which runs as user and group mapred:hadoop by default, cannot access
the aggregated logs.

I'm not sure what is a good way of fixing this. 

There does not seem to be a way to fix this behavior through the configuration. While run-as
groups can be specified, they do not seem to affect the created directories.

LogAggregationService should probably use the Nodemanager's run-as user AND group (which default
to yarn:hadoop) to create /tmp/logs rather than leave the group unchanged.

On the other hand, the user and app dirs should better be created with the group unchanged
(i.e., hadoop).






--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message