Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Date: Mon, 21 Feb 2011 23:16:39 +0000 (UTC)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: 
 <2078789276.6851.1298330199650.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (MAPREDUCE-279) Map-Reduce 2.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12997600#comment-12997600 ] 

Arun C Murthy commented on MAPREDUCE-279:
-----------------------------------------

bq. With ZooKeeper in there, I wonder how much of the communicaton now uses ZooKeeper watches for efficiency and low latency.

Scott - We seriously considered this, but had to continue to use Hadoop RPC for a couple of reasons:
a) Mahadev, our resident ZK (and the new ResourceManager) expert, was very vary of using ZK watches for scalability reasons. Consider a 10k node cluster with 25-30 containers per node and 10k running jobs - we'd need at least 10k * 10k watches which is a *lot* for ZK
b) Security on ZK is still largely unknown, eventually ZK will get there but we'd need a lot of work to do for delegation tokens etc. since we can't do kerberos everywhere.

Having said that...

bq. In any case, the new architecture is decoupled and it should be much easier to make enhancements with this separation.

Exactly. This is something we should definitely re-visit in a subsequent release. Hopefully that makes sense, thanks!

> Map-Reduce 2.0
> --------------
>
>                 Key: MAPREDUCE-279
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-279
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker, tasktracker
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 0.23.0
>
>
> Re-factor MapReduce into a generic resource scheduler and a per-job, user-defined component that manages the application execution. 

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira