Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MailScanner-NULL-Check: 1215607743.84345@pCL90SZ6VKTqZAGjQsf+kQ
Message-ID: <486B7949.8010103@apache.org>
Date: Wed, 02 Jul 2008 13:49:13 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.14 (X11/20080421)
MIME-Version: 1.0
To: core-dev@hadoop.apache.org
Subject: Re: porting HDFS to Zookeeper??
References: <811fc3830806301336r393d3b61k3f9e43b5ef7909fd@mail.gmail.com>
	 <48696E79.5030009@yahoo-inc.com>
 <811fc3830807010141t76246fd0m9205c7dfe87c7069@mail.gmail.com>
In-Reply-To: <811fc3830807010141t76246fd0m9205c7dfe87c7069@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Sangmin Lee wrote:
> Thank you for the replies.
> I meant porting HDFS to Zookeeper to make namenode fault-tolerant and highly
> available rather than the simple usage of Zookeeper. Basically by storing
> namespace metadata in Zookeeper, we may be able to get highly available
> replicated namenodes. So I meant using Zookeeper as frontend of HDFS. Anyway
> it seems to me that there has been no actual work going on regarding this,
> isn't it? :-)
> 
> -sangmin


I think you've just volunteered.

@work we use our own tuple space thing, Anubis:
http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components/anubis/doc/

it lets you do the election/voting stuff to see who is in charge in an 
HA cluster, though to deal with a partition event robustly is hard; you 
need somebody that really understands the maths behind HA distributed 
computing. I am not in that category, so we are lucky there are others 
nearby who do, but it implies there is a block of code I am very 
reluctant to approach.

In fact, its a bit like security. just like it is easy to do an insecure 
system, but very hard to do a secure one, its easy to be 
low-availability, really, really hard to be HA.

-steve