hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mahadev konar (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-279) Map-Reduce 2.0
Date Mon, 28 Feb 2011 22:28:36 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000576#comment-13000576

Mahadev konar commented on MAPREDUCE-279:

 With respect your comments on ResourceManager/ZooKeeper/RPC:
We intend to take it slow with ZooKeeper, initially the intention is to put just the allocations
(what each job/application is allocated in ZooKeeper, this is mainly for ResourceManager and
Application Master restart). I am not really in favor of  using ZK notification for getting
rid of RPC's. For the scale we are talking abt, the "first to get the work will take it" approach
will cause herd affect and will definitely be a cause for concern. I think ZK can be used
much more than what we have proposed but itll be gradual process to see what all we can offload
to ZK.

I am pretty hesitant to put RPC load onto ZK and use it as a workload queue for something
like this. 

> Map-Reduce 2.0
> --------------
>                 Key: MAPREDUCE-279
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-279
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker, tasktracker
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 0.23.0
> Re-factor MapReduce into a generic resource scheduler and a per-job, user-defined component
that manages the application execution. 

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message