hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-512) Log aggregation root directory check is more expensive than it needs to be
Date Wed, 29 May 2013 19:38:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13669604#comment-13669604
] 

Vinod Kumar Vavilapalli commented on YARN-512:
----------------------------------------------

I was going to comment on it. Why is the verifyAndCreate method getting called for every app?
Shouldn't this be only once? This could explain one part of the gazillions connections we
are creating to name-node during log-aggregation.
                
> Log aggregation root directory check is more expensive than it needs to be
> --------------------------------------------------------------------------
>
>                 Key: YARN-512
>                 URL: https://issues.apache.org/jira/browse/YARN-512
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.0.5-beta
>            Reporter: Jason Lowe
>            Assignee: Maysam Yabandeh
>            Priority: Minor
>             Fix For: 2.0.5-beta
>
>         Attachments: YARN-512.patch
>
>
> The log aggregation root directory check first does an {{exists}} call followed by a
{{getFileStatus}} call.  That effectively stats the file twice.  It should just use {{getFileStatus}}
and catch {{FileNotFoundException}} to handle the non-existent case.
> In addition we may consider caching the presence of the directory rather than checking
it each time a node aggregates logs for an application.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message