hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doğacan Güney" <doga...@gmail.com>
Subject Re: JavaSpaces (Blitz?) and hadoop - comparison?
Date Fri, 02 Mar 2007 11:15:31 GMT

On 3/2/07, Dan Creswell <dan.creswell@lonecrusader.co.uk> wrote:
> Nigel Daley wrote:
> > One more difference...
> >
> > Being that JavaSpaces is a Jini service, its host/port can be
> > dynamically discovered (and rediscovered else where if it fails) at
> > run time by clients.
> > OTOH, Hadoop servers and clients are currently pre-configured with
> > necessary host/ports.
> >
> Indeed - I was toying with doing something about removing this
> pre-configuration - worthwhile?

Here is a description of how Google does this (As far as I understand
it) (Everyone here probably knows this but anyway). Google uses Chubby
to locate GFS and Bigtable masters (also probably jobtracker master
too). In a data center there are typically 5 chubby instances and
their locations are stored in DNS. A client reads their location from
DNS and then traverses them to locate the Chubby master. After
locating it, the client reads a predefined file in Chubby to locate
the GFS/Bigtable/MR master. If, say, a GFS namenode dies, a GFS
datanode becomes the namenode and updates the file in Chubby. If the
Chubby master dies, a Chubby slave becomes the master.  If a client
tries to connect to the old Chubby master, the connection will timeout
and client will retraverse all Chubby instances to find out which is
the master.

[snipped the rest]

Doğacan Güney
View raw message