hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3902) MR AM should reuse containers for map tasks, there-by allowing fine-grained control on num-maps for users without need for CombineFileInputFormat etc.
Date Tue, 04 Sep 2012 18:55:11 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447943#comment-13447943
] 

Siddharth Seth commented on MAPREDUCE-3902:
-------------------------------------------

Thanks for the help with this JIRA.
bq. because MRAppMaster in container-reuse implementation has the feature to monitor whether
the running tasks on the containers are "the last task at a machine or not", for the purpose
of exiting JVMs on containers, as you know.
That will definitely be simpler to achieve with the container-reuse AM, with nodes already
tracking container information. Last task on a node can be figured out relatively easily by
the scheduler. It is, however, also possible with the current AM, and several bits like the
decision on when to run the combiner - should be a straight forward port to the reuse-AM.
IAC, it'll be good to get the re-use AM into trunk fast. Looking forward to the updates on
4502 and 4525. 

                
> MR AM should reuse containers for map tasks, there-by allowing fine-grained control on
num-maps for users without need for CombineFileInputFormat etc.
> ------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3902
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3902
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: applicationmaster, mrv2
>            Reporter: Arun C Murthy
>            Assignee: Siddharth Seth
>         Attachments: MAPREDUCE-3902.2.patch, MAPREDUCE-3902.patch
>
>
> The MR AM is now in a great position to reuse containers across (map) tasks. This is
something similar to JVM re-use we had in 0.20.x, but in a significantly better manner:
> # Consider data-locality when re-using containers
> # Consider the new shuffle - ensure that reduces fetch output of the whole container
at once (i.e. all maps)  : MAPREDUCE-4525 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message