hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-71) Ensure/confirm that the NodeManager cleans up local-dirs on restart
Date Tue, 12 Mar 2013 02:33:14 GMT

    [ https://issues.apache.org/jira/browse/YARN-71?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13599622#comment-13599622

Siddharth Seth commented on YARN-71:

Thanks for the patches Xuan. Couple of comments on the latest one (71-6.txt)

- The ResourceLocalizationService is already creating a FileContxt instance - that can be
- The rename/exception code is the same across the 3 top level dirs and can move into a function.
- The patch needs some formatting fixes (missing spaces after "+", "," etc)
- Haven't looked at the unit test yet, but I don't think it belongs in TestNodeStatusUpdater.

In terms of the approach of moving dirs and scheduling them for background deletion - this
seems reasonable to me. Once concern I have though is that this could cause some load on the
NMs after they're restarted (especially on systems with a large dist cache and disks). This
effectively ends up affecting performance. Anyone else have thoughts on this ? Maybe background
/ inline deletion should be configurable ?
> Ensure/confirm that the NodeManager cleans up local-dirs on restart
> -------------------------------------------------------------------
>                 Key: YARN-71
>                 URL: https://issues.apache.org/jira/browse/YARN-71
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Xuan Gong
>            Priority: Critical
>         Attachments: YARN-71.1.patch, YARN-71.2.patch, YARN-71.3.patch, YARN.71.4.patch,
YARN-71.5.patch, YARN-71.6.patch
> We have to make sure that NodeManagers cleanup their local files on restart.
> It may already be working like that in which case we should have tests validating this.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message