hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2261) YARN should have a way to run post-application cleanup
Date Tue, 08 Jul 2014 06:35:34 GMT

    [ https://issues.apache.org/jira/browse/YARN-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054571#comment-14054571
] 

Vinod Kumar Vavilapalli commented on YARN-2261:
-----------------------------------------------

The proposal here is to have a YARN application-level cleanup container that runs only as
the last thing for an application in the cluster.
 - In a way, we already have this today, as we let AM's hang around for a while (by default
- 10mins) *after* unregister - this feature makes it explicit.
 - For those who have lived in this space around for a while, this is akin to MR job-cleanup.
 - This feature lets apps submit a separate container-launch-context for cleanup, one that
is only run after the app is done for real.
 - Clearly, it will
    -- be optional.
    -- Have timeouts on how much time it can take to finish (default, overridable, and upper
limit. Default = today's time for AMs to exit after unregister?)
    -- Have resource requests limits like usual
    -- May have its own retries (Cleanup failure != Application failure as today?)

Some challenges
 - Cleanup container may not get resources because cluster may have gotten busy after the
final AM exit. Solution is to reserve (part of) resources used by the last AM for use by the
cleanup container

> YARN should have a way to run post-application cleanup
> ------------------------------------------------------
>
>                 Key: YARN-2261
>                 URL: https://issues.apache.org/jira/browse/YARN-2261
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: resourcemanager
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
>
> See MAPREDUCE-5956 for context. Specific options are at https://issues.apache.org/jira/browse/MAPREDUCE-5956?focusedCommentId=14054562&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14054562.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message