hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-279) Map-Reduce 2.0
Date Mon, 21 Feb 2011 23:16:39 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12997600#comment-12997600

Arun C Murthy commented on MAPREDUCE-279:

bq. With ZooKeeper in there, I wonder how much of the communicaton now uses ZooKeeper watches
for efficiency and low latency.

Scott - We seriously considered this, but had to continue to use Hadoop RPC for a couple of
a) Mahadev, our resident ZK (and the new ResourceManager) expert, was very vary of using ZK
watches for scalability reasons. Consider a 10k node cluster with 25-30 containers per node
and 10k running jobs - we'd need at least 10k * 10k watches which is a *lot* for ZK
b) Security on ZK is still largely unknown, eventually ZK will get there but we'd need a lot
of work to do for delegation tokens etc. since we can't do kerberos everywhere.

Having said that...

bq. In any case, the new architecture is decoupled and it should be much easier to make enhancements
with this separation.

Exactly. This is something we should definitely re-visit in a subsequent release. Hopefully
that makes sense, thanks!

> Map-Reduce 2.0
> --------------
>                 Key: MAPREDUCE-279
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-279
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker, tasktracker
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 0.23.0
> Re-factor MapReduce into a generic resource scheduler and a per-job, user-defined component
that manages the application execution. 

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message