Mailing-List: contact mapreduce-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of sharad.apache@gmail.com
 designates 209.85.216.43 as permitted sender)
References: <000601cc6e2c$5bc909a0$135b1ce0$%kotha@huawei.com>
 <CAMEEY-L_X0SY-JPKGNbprncK5apgOdGLi1+ALaRtGCwFDaR3SA@mail.gmail.com>
Message-Id: <59767724-8BD1-4994-943E-153A2E9E21FB@gmail.com>
From: Sharad <sharad.apache@gmail.com>
To: "mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>
In-Reply-To: 
 <CAMEEY-L_X0SY-JPKGNbprncK5apgOdGLi1+ALaRtGCwFDaR3SA@mail.gmail.com>
Content-Type: text/plain;
	charset=us-ascii;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (iPhone Mail 7E18)
Subject: Re: Rationale behind Event based design of Next Gen Mapreduce
 components.
Date: Thu, 8 Sep 2011 20:11:05 +0530
Cc: "mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>,
 "venugopalarao.kotha@huawei.com" <venugopalarao.kotha@huawei.com>

Thanks vinod. Just to add it also gives better parallelism and hence  
scalability. OTOH with sync model with so many moving parts it is  
really very hard to design and maintain fine-grained locking.

Sent from my iPhone

On Sep 8, 2011, at 7:33 PM, Vinod Kumar Vavilapalli <vinodkv@hortonworks.com 
 > wrote:

> This question needs a long answer and a bit of documentation+wiki  
> pages
> explaining the whole thing which I am working on.
>
> For now, I'll give you a short answer.
>
> If you have worked long enough on MRV1, particularly JobTracker and
> TaskTracker, you will know the complexity of the code w.r.t component
> interactions, synchronization, method calls in and out, management  
> of state
> via ENUMs etc. That was quite some maintenance nightmare if you ask  
> me.
>
> The event model along with the state machines are an effort to  
> manage that
> complexity better.
>
> There is one slide "Event Model in YARN" in Sharad's presentation  
> attached
> on MAPREDUCE-279:
> https://issues.apache.org/jira/secure/attachment/12485267/hadoop_contributors_meet_07_01_2011.pdfthat
> can be served as a starter.
>
> Thanks,
> +Vinod
>
> On Thu, Sep 8, 2011 at 7:06 PM, Venu Gopala Rao <
> venugopalarao.kotha@huawei.com> wrote:
>
>> Hi All,
>>
>>
>>
>>   I am going through the Next Gen mapReduce code base. Unlike MRV1  
>> all the
>> components work based on event dispatching/consumption. Some times  
>> I see it
>> is a long list of events :) .
>>
>>
>>
>>  I would like to understand, is there any specific reason behind this
>> event based design?
>>
>>
>>
>> Regards
>>
>> Venu
>>
>>
>>
>>
>>
>>
>>
>>