hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Venu Gopala Rao <venugopalarao.ko...@huawei.com>
Subject RE: Rationale behind Event based design of Next Gen Mapreduce components.
Date Fri, 09 Sep 2011 04:34:47 GMT
Thanks Sharad and Vinod.

-----Original Message-----
From: Sharad [mailto:sharad.apache@gmail.com] 
Sent: Thursday, September 08, 2011 8:11 PM
To: mapreduce-dev@hadoop.apache.org
Cc: mapreduce-dev@hadoop.apache.org; venugopalarao.kotha@huawei.com
Subject: Re: Rationale behind Event based design of Next Gen Mapreduce
components.

Thanks vinod. Just to add it also gives better parallelism and hence  
scalability. OTOH with sync model with so many moving parts it is  
really very hard to design and maintain fine-grained locking.

Sent from my iPhone

On Sep 8, 2011, at 7:33 PM, Vinod Kumar Vavilapalli <vinodkv@hortonworks.com

 > wrote:

> This question needs a long answer and a bit of documentation+wiki  
> pages
> explaining the whole thing which I am working on.
>
> For now, I'll give you a short answer.
>
> If you have worked long enough on MRV1, particularly JobTracker and
> TaskTracker, you will know the complexity of the code w.r.t component
> interactions, synchronization, method calls in and out, management  
> of state
> via ENUMs etc. That was quite some maintenance nightmare if you ask  
> me.
>
> The event model along with the state machines are an effort to  
> manage that
> complexity better.
>
> There is one slide "Event Model in YARN" in Sharad's presentation  
> attached
> on MAPREDUCE-279:
>
https://issues.apache.org/jira/secure/attachment/12485267/hadoop_contributor
s_meet_07_01_2011.pdfthat
> can be served as a starter.
>
> Thanks,
> +Vinod
>
> On Thu, Sep 8, 2011 at 7:06 PM, Venu Gopala Rao <
> venugopalarao.kotha@huawei.com> wrote:
>
>> Hi All,
>>
>>
>>
>>   I am going through the Next Gen mapReduce code base. Unlike MRV1  
>> all the
>> components work based on event dispatching/consumption. Some times  
>> I see it
>> is a long list of events :) .
>>
>>
>>
>>  I would like to understand, is there any specific reason behind this
>> event based design?
>>
>>
>>
>> Regards
>>
>> Venu
>>
>>
>>
>>
>>
>>
>>
>>


Mime
View raw message