hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Carey (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-279) Map-Reduce 2.0
Date Mon, 21 Feb 2011 22:17:38 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12997589#comment-12997589

Scott Carey commented on MAPREDUCE-279:

Good stuff!

Does the NodeManager communicate to the ResourceManager similar to now (ping -> response
RPC)?   I ask because some of the bottlenecks and complexities now are due to this style of
RPC.  I've changed a couple systems in the past from ping -> response to register ->
callback in the past and these became more efficient and the code became simpler.  With ZooKeeper
in there, I wonder how much of the communicaton now uses ZooKeeper watches for efficiency
and low latency.

When a Job starts up in the ApplicationMaster, does it have to wait for pings to get resources
from the scheduler?  Or is the data all there in ZK, so that ramp-up times for jobs is much
faster and resource reassignment for jobs with short lived tasks isn't completely throttled
by the rate of pings?

In any case, the new architecture is decoupled and it should be much easier to make enhancements
with this separation.

> Map-Reduce 2.0
> --------------
>                 Key: MAPREDUCE-279
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-279
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker, tasktracker
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 0.23.0
> Re-factor MapReduce into a generic resource scheduler and a per-job, user-defined component
that manages the application execution. 

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message