hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sandy Ryza (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-512) Log aggregation root directory check is more expensive than it needs to be
Date Wed, 29 May 2013 20:08:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13669650#comment-13669650

Sandy Ryza commented on YARN-512:

[~tucu00] and I talked about this when working on YARN-24.  An earlier version of the patch
didn't have this, but our thought was that verifying log dir permissions was worth an RPC
or two per application.
> Log aggregation root directory check is more expensive than it needs to be
> --------------------------------------------------------------------------
>                 Key: YARN-512
>                 URL: https://issues.apache.org/jira/browse/YARN-512
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.0.5-beta
>            Reporter: Jason Lowe
>            Assignee: Maysam Yabandeh
>            Priority: Minor
>             Fix For: 2.0.5-beta
>         Attachments: YARN-512.patch
> The log aggregation root directory check first does an {{exists}} call followed by a
{{getFileStatus}} call.  That effectively stats the file twice.  It should just use {{getFileStatus}}
and catch {{FileNotFoundException}} to handle the non-existent case.
> In addition we may consider caching the presence of the directory rather than checking
it each time a node aggregates logs for an application.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message