hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3162) Separate application-init and container-init event types in NM's ApplicationImpl FSM
Date Tue, 11 Oct 2011 08:06:29 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13124791#comment-13124791

Chris Douglas commented on MAPREDUCE-3162:

bq. The issue here is that "startLocalizer" not only handles localizing resources, but also
mkdirs/chowns the appCachedir, log dirs, etc. So we need to have some way of making sure that
stuff happens once (preferably exactly once), in addition to being lazy about launching the
actual localizer subprocess. Does that make any sense?

Sure. We went back and forth on this, but the operations saved by adding a fast path for init
are pretty minor (one hopes) given the cost of a container. Asserting every precondition for
container launch _on_ every container launch is often redundant (particularly for MR), but
making it idempotent seemed cleaner than adding all the logic to avoid it. That section also
has a lot of rough patches; it's a minor crime for that neighborhood.

bq. We definitely need to move the creation of app-dirs etc out of resource-localization.

It's a pretty inexpensive operation and pushing the dir creation into the resource localization
puts all that into a single fork for the LCE (required root privs to create the userdir).
But yea, it's usually waste.
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-3162
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: mr-3162.txt
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container
initialization. Only on the first one does it really mean to init the application, whereas
all subsequent events are for specific containers. This JIRA is to separate the events into
INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent
for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message