hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: Decoupling Intertracker protocol
Date Sat, 25 Apr 2009 17:06:50 GMT
I think that this kind of strategy could easily scale to clusters of
thousands of nodes and possibly much more (if you have a good communication
substrate).

I looked at the comet documentation and it really looks like they went a
long ways to make sure that they are a pure p2p coordination layer.  That is
definitely an interesting research project, but it makes it much, much less
interesting in terms of realistic systems, if only because the resulting
system is so much more complex.

This sounds like an academic project, though, so comparing a p2p
coordination layer with state of the commercial art alternatives is probably
a good thing.  I would recommend that if you want a very interesting result,
you should implement your coordination with BOTH zookeeper and comet and
figure out what the differences are.

On Fri, Apr 24, 2009 at 11:47 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:

> I am not sure if this strategy is good for large scale Map_Reduce
>> applications. But it might work well for small scale Map-reduce jobs.
>>
>> More Information on the co-ordination space that I want to use can be
>> found
>> here : http://www.caip.rutgers.edu/~zhljenny/comet.htm<http://www.caip.rutgers.edu/%7Ezhljenny/comet.htm>.
>>
>
>


-- 
Ted Dunning, CTO
DeepDyve

111 West Evelyn Ave. Ste. 202
Sunnyvale, CA 94086
www.deepdyve.com
858-414-0013 (m)
408-773-0220 (fax)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message