Return-Path: 
 <mapreduce-dev-return-3582-apmail-hadoop-mapreduce-dev-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 06B136614
	for <apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org>;
 Tue, 14 Jun 2011 19:30:08 +0000 (UTC)
Received: (qmail 17071 invoked by uid 500); 14 Jun 2011 19:30:07 -0000
Delivered-To: apmail-hadoop-mapreduce-dev-archive@hadoop.apache.org
Received: (qmail 16922 invoked by uid 500); 14 Jun 2011 19:30:07 -0000
Mailing-List: contact mapreduce-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:mapreduce-dev-help@hadoop.apache.org>
List-Unsubscribe: <mailto:mapreduce-dev-unsubscribe@hadoop.apache.org>
List-Post: <mailto:mapreduce-dev@hadoop.apache.org>
List-Id: <mapreduce-dev.hadoop.apache.org>
Reply-To: mapreduce-dev@hadoop.apache.org
Delivered-To: mailing list mapreduce-dev@hadoop.apache.org
Received: (qmail 16914 invoked by uid 99); 14 Jun 2011 19:30:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 19:30:07 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 216.145.54.172 is neither permitted
 nor denied by domain of acm@yahoo-inc.com)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 19:30:02 +0000
Received: from [192.168.1.9] (vpn-client-100-19.eglbp.corp.yahoo.com
 [10.66.100.19])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5EJTQ1Y069722
	for <mapreduce-dev@hadoop.apache.org>; Tue, 14 Jun 2011 12:29:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1308079769;
	bh=B9sLwB9nWjfllCIl9qbC6wm0RTD5khvTum6iVWwkgRE=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=g3SyUIDYDKyWYtkS/vCRep9G6tzY+Qupm9ir0u/jBXgxoST2vYYh+F1940iCpPQxO
	 nPP8QDH9yAdvvji3aM51S+4DASuJN186q8rkoKV1duoUYkXIGoHsiM/sqGpZipyze5
	 DPaMfGUmQgGZwuhlxHL4cNU9gRKsdfBq5EwJuH3g=
Message-Id: <4B979539-60E1-4A79-A31C-D5F72CC1F20E@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>
In-Reply-To: <BANLkTimbB9GV-VL00ct+-8cQOyMLZYZmSQ@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Queries on MRv2
Date: Wed, 15 Jun 2011 00:59:26 +0530
References: <BANLkTimbB9GV-VL00ct+-8cQOyMLZYZmSQ@mail.gmail.com>
X-Mailer: Apple Mail (2.936)


On Jun 14, 2011, at 6:31 PM, Praveen Sripati wrote:

> Hi,
>
> I have gone through MapReduce NextGen Blog entries and JIRA and have  
> the
> following queries
>
>>> There is a single API between the Scheduler and the  
>>> ApplicationMaster:
>
>>> (List <Container> newContainers, List <ContainerStatus>
> containerStatuses) allocate (List <ResourceRequest> ask,  
> List<Container>
> release)
>
>>> The AM ask for specific resources via a list of ResourceRequests  
>>> (ask)
> and releases unnecessary Containers which were allocated by the  
> Scheduler.
>
>>> The response contains a list of newly allocated Containers and the
> statuses of application-specific Containers that completed since the
> previous interaction between the AM and the RM.
>
> Q) If split-0 is is available in host1, host2 and host3, can
> ApplicationMaster request a scheduler for a container on host1 or  
> host2 or
> host3? This way the scheduler can allocate the resources more  
> effectively.
>

Yes, absolutely.

> Q) In a cluster there might be nodes of different capacities, how  
> will the
> scheduler know that a particular node has 4 GB and another has 16 GB  
> RAM
> before allocating the resources to the ApplicationMaster?
>

The NodeManager informs the RM about its capabilities on registration.  
The RM allocates appropriate resources to the AM(s).

> Q) Are the unnecessary containers (List<Container> release) in the  
> request
> released by the ApplicationMaster the ones rejected by the  
> ApplicationMaster
> or those on which the map/reduce tasks have been completed?
>

Only unused ones.

> Q) What does the following in the response contain - "List  
> <ContainerStatus>
> containerStatuses"?
>

Status for completed completed containers.

> Q) Once the ApplicationMaster gets the list of the new containers  
> from the
> Scheduler, what is the interaction between the ApplicationMaster and  
> the
> Node Manager? Will the ApplicationMaster ask the Node Manager on the
> different nodes to launch/monitor the map/reduce tasks in those  
> containers?
>

No, the AM directly monitors the containers via an application- 
specific protocol.

For MR applications we use TaskUmbilicalProtocol.

The NM just monitors the unix process and informs the RM on exit of  
the unix process.

> Q) Does the Scheduler ask the Node Manager to create the containers  
> on the
> different nodes?

No, the Scheduler allocates them to the respective AMs who then launch  
the container by talking to the NM.

The NM can securely verify the authenticity of the 'container launch'  
request, including the resources allocated to the container.

>
>>> The resource requests are also aggregated by racks and then by the
> special any (*) for all containers. All resource requests are  
> subject to
> change via the delta protocol.
>
> Q) Does (*) mean that the ApplicationMaster is OK with a container  
> in any
> rack/host? This might be applicable for Reduce tasks.
>
Yes.

Hope this helps.

Arun