hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-742) Log aggregation causes a lot of redundant setPermission calls
Date Mon, 03 Jun 2013 22:12:20 GMT

     [ https://issues.apache.org/jira/browse/YARN-742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Jason Lowe updated YARN-742:

    Attachment: YARN-742.patch

Patch to walk back up the app log dir path to check for the existence of a directory before
blindly proceeding to create it.  It gets out early if it finds a path that exists.

In addition to the unit test, I manually tested this on a single-node cluster verifying the
directories are created iff they are missing and permissions are set iff necessary due to
a too-restrictive umask.
> Log aggregation causes a lot of redundant setPermission calls
> -------------------------------------------------------------
>                 Key: YARN-742
>                 URL: https://issues.apache.org/jira/browse/YARN-742
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 0.23.7, 2.0.4-alpha
>            Reporter: Kihwal Lee
>            Assignee: Jason Lowe
>         Attachments: YARN-742.patch
> In one of our clusters, namenode RPC is spending 45% of its time on serving setPermission
calls. Further investigation has revealed that most calls are redundantly made on /mapred/logs/<user>/logs.
Also mkdirs calls are made before this.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message