hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Greg Roelofs (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2405) MR-279: Implement uber-AppMaster (in-cluster LocalJobRunner for MRv2)
Date Sat, 26 Mar 2011 05:21:05 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011568#comment-13011568

Greg Roelofs commented on MAPREDUCE-2405:

As with MAPREDUCE-1220's "UberTask", which essentially runs small jobs sequentially within
a single Task (and therefore in a single JVM), the MRv2 version does so within an "UberAppMaster"--which
is really just the regular MRAppMaster with two container services overridden (allocator,
launcher).  The analogue to UberTask's run() method is LocalContainerLauncher's SubtaskRunner.run(),
which hooks into the state machines and executes the subtasks sequentially.

This design is much cleaner than that in MR-1220 since the subtasks are "real" and can communicate
directly with external entities, and the uber-AM is the regular AM and requires no special
UI handling (though we'll want to flag uberized AMs and jobs in the UI somehow).  UberTask,
on the other hand, had to translate its subtasks' status-updates and was very awkward to integrate
with the UI (is it a ReduceTask or an UberTask? -- the cascade of required changes was huge
and never completed).

Kudos go to Sharad Agarwal for this design.

> MR-279: Implement uber-AppMaster (in-cluster LocalJobRunner for MRv2)
> ---------------------------------------------------------------------
>                 Key: MAPREDUCE-2405
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2405
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv2
>            Reporter: Mahadev konar
>            Assignee: Greg Roelofs
>             Fix For: 0.23.0
> "Port" MAPREDUCE-1220 to MRv2.  This is an optimization for small jobs wherein all tasks
run on the same node in the same JVM/container.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message