hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sharad <sharad.apa...@gmail.com>
Subject Re: Rationale behind Event based design of Next Gen Mapreduce components.
Date Thu, 08 Sep 2011 14:41:05 GMT
Thanks vinod. Just to add it also gives better parallelism and hence  
scalability. OTOH with sync model with so many moving parts it is  
really very hard to design and maintain fine-grained locking.

Sent from my iPhone

On Sep 8, 2011, at 7:33 PM, Vinod Kumar Vavilapalli <vinodkv@hortonworks.com 
 > wrote:

> This question needs a long answer and a bit of documentation+wiki  
> pages
> explaining the whole thing which I am working on.
>
> For now, I'll give you a short answer.
>
> If you have worked long enough on MRV1, particularly JobTracker and
> TaskTracker, you will know the complexity of the code w.r.t component
> interactions, synchronization, method calls in and out, management  
> of state
> via ENUMs etc. That was quite some maintenance nightmare if you ask  
> me.
>
> The event model along with the state machines are an effort to  
> manage that
> complexity better.
>
> There is one slide "Event Model in YARN" in Sharad's presentation  
> attached
> on MAPREDUCE-279:
> https://issues.apache.org/jira/secure/attachment/12485267/hadoop_contributors_meet_07_01_2011.pdfthat
> can be served as a starter.
>
> Thanks,
> +Vinod
>
> On Thu, Sep 8, 2011 at 7:06 PM, Venu Gopala Rao <
> venugopalarao.kotha@huawei.com> wrote:
>
>> Hi All,
>>
>>
>>
>>   I am going through the Next Gen mapReduce code base. Unlike MRV1  
>> all the
>> components work based on event dispatching/consumption. Some times  
>> I see it
>> is a long list of events :) .
>>
>>
>>
>>  I would like to understand, is there any specific reason behind this
>> event based design?
>>
>>
>>
>> Regards
>>
>> Venu
>>
>>
>>
>>
>>
>>
>>
>>

Mime
View raw message