hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-71) Ensure/confirm that the NodeManager cleans up local-dirs on restart
Date Tue, 12 Mar 2013 02:33:14 GMT

    [ https://issues.apache.org/jira/browse/YARN-71?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13599622#comment-13599622
] 

Siddharth Seth commented on YARN-71:
------------------------------------

Thanks for the patches Xuan. Couple of comments on the latest one (71-6.txt)

- The ResourceLocalizationService is already creating a FileContxt instance - that can be
used.
- The rename/exception code is the same across the 3 top level dirs and can move into a function.
- The patch needs some formatting fixes (missing spaces after "+", "," etc)
- Haven't looked at the unit test yet, but I don't think it belongs in TestNodeStatusUpdater.

In terms of the approach of moving dirs and scheduling them for background deletion - this
seems reasonable to me. Once concern I have though is that this could cause some load on the
NMs after they're restarted (especially on systems with a large dist cache and disks). This
effectively ends up affecting performance. Anyone else have thoughts on this ? Maybe background
/ inline deletion should be configurable ?
                
> Ensure/confirm that the NodeManager cleans up local-dirs on restart
> -------------------------------------------------------------------
>
>                 Key: YARN-71
>                 URL: https://issues.apache.org/jira/browse/YARN-71
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Xuan Gong
>            Priority: Critical
>         Attachments: YARN-71.1.patch, YARN-71.2.patch, YARN-71.3.patch, YARN.71.4.patch,
YARN-71.5.patch, YARN-71.6.patch
>
>
> We have to make sure that NodeManagers cleanup their local files on restart.
> It may already be working like that in which case we should have tests validating this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message