hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: Decoupling Intertracker protocol
Date Sat, 25 Apr 2009 17:02:10 GMT
This sounds like a perfect application of Zookeeper.


On Fri, Apr 24, 2009 at 11:47 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:

> On Apr 22, 2009, at 7:17 PM, Samprita Hegde wrote:
>
>
> Currently all the information seems to be sent via the heartbeat message in
>> the InterTracker Protocol. Is there a way where I can decouple only some
>> part of heartbeat message and put it on to the space? (Especially the
>> TaskCompletionEvent and TaskStatus). Using this the task completion events
>> can be exchanged directly among the Task Trackers adn not through the Job
>> Tracker.
>>
>
> You would need to fix the TaskTrackers to update the shared-space rather
> than send it to the JobTracker.
>
> One pertinent point to remember is that you do need some global arbitration
> for e.g. deciding which among the concurrently successful speculative tasks
> are to be declared as 'successful' etc.
>



-- 
Ted Dunning, CTO
DeepDyve

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message