Mailing-List: contact zookeeper-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: zookeeper-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:return-path:x-originalarrivaltime;
	b=JVE+h/gF9RxYCTVjz7SlX6ZupMcnvKnuBnXCUps9eJfP2zyvHU3jFrJc5JQI2QNO
User-Agent: Microsoft-Entourage/12.24.0.100205
Date: Mon, 26 Jul 2010 09:12:31 -0700
Subject: Re: node symlinks
From: Mahadev Konar <mahadev@yahoo-inc.com>
To: <zookeeper-user@hadoop.apache.org>
Message-ID: <C87301FF.3BB36%mahadev@yahoo-inc.com>
Thread-Topic: node symlinks
Thread-Index: Acss3VluPktANlKQ1UCkIhehOJnTsQ==
In-Reply-To: <4C4B5DA4.9030401@vrijheid.net>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit

HI Maarteen,
  Can you elaborate on your use case of ZooKeeper? We currently don't have
any symlinks feature in zookeeper. The only way to do it for you would be a
client side hash/lookup table that buckets data to different zookeeper
servers. 

Or you could also store this hash/lookup table in one of the zookeeper
clusters. This lookup table can then be cached on the client side after
reading it once from zookeeper servers.

Thanks
mahadev


On 7/24/10 2:39 PM, "Maarten Koopmans" <maarten@vrijheid.net> wrote:

> Yes, I thought about Cassandra or Voldemort, but I need ZKs guarantees
> as it will provide the file system hierarchy to a flat object store so I
> need locking primitives and consistency. Doing that on top of Voldemort
> will give me a scalable version of ZK, but just slower. Might as well
> find a way to scale across ZK clusters.
> 
> Also, I want to be able to add clusters as the number of nodes grows.
> Note that the #nodes will grow with the #users of the system, so the
> clusters can grow sequentially, hence the symlink idea.
> 
> --Maarten
> 
> On 07/24/2010 11:12 PM, Ted Dunning wrote:
>> Depending on your application, it might be good to simply hash the node name
>> to decide which ZK cluster to put it on.
>> 
>> Also, a scalable key value store like Voldemort or Cassandra might be more
>> appropriate for your application.  Unless you need the hard-core guarantees
>> of ZK, they can be better for large scale storage.
>> 
>> On Sat, Jul 24, 2010 at 7:30 AM, Maarten Koopmans<maarten@vrijheid.net>wrote:
>> 
>>> Hi,
>>> 
>>> I have a number of nodes that will grow larger than one cluster can hold,
>>> so I am looking for a way to efficiently stack clusters. One way is to have
>>> a zookeeper node "symlink" to another cluster.
>>> 
>>> Has anybody ever done that and some tips, or alternative approaches?
>>> Currently I use Scala, and traverse zookeeper trees by proper tail
>>> recursion, so adapting the tail recursion to process "symlinks" would be my
>>> approach.
>>> 
>>> Bst, Maarten
>>> 
>> 
>