hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "ZooKeeper/PartitionedZookeeper" by FlavioJunqueira
Date Wed, 19 Nov 2008 16:24:01 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The following page has been changed by FlavioJunqueira:

New page:
== Paritioned Zookeeper ==

=== 10,000 ft view ===

Our main goal is to enable write throughput scalability through partitions in ZooKeeper. The
overall idea is to split a set of zookeeper servers into overlapping ensembles, and have each
partition handle a portion of the ZooKeeper state. By having distinct ensembles handling different
portions of the state, we end up relaxing the ordering guarantees that we have with plain
ZooKeeper. To overcome this problem, we provide an abstraction that we call "containers".
Containers are subtrees of znodes which require that all update operations are ordered as
with plain ZooKeeper.   

=== Containers ===

Containers are subtrees of znodes, and the root of a container is not necessarily the root
of a ZooKeeper tree. Upon the creation of a node, we state whether to create a new container
for that node or not.

=== Changes to the API ===

The only change we envision is an extra parameter on create that tells whether to create a
new container or not for the new node. All other operations should be the same. 

=== Internal changes ===

Internally, we will require more changes:

 1. '''Routing''': We need a mechanism to route requests to the correct ensemble. We can perform
it in a distributed fashion, as with DHTs, or we can have one ZooKeeper ensemble responsible
for mapping prefixes to ensembles;
 1. '''Containers''': A ZooKeeper server with this approach has to store and handle requests
for a set of containers. Such a set may contain containers from different partitions. Handling
different subsets for different partitions does not necessarily imply having multiple instances
of the ZooKeeper server on a single machine because containers are disjoint by definition
and they can be operated upon in parallel. We just have to make sure that we identify a container
correctly when executing a ZooKeeper operation; 
 1. '''Failure and Recovery''': Upon failure of a server, the immediate neighbor of that server
should take over the position of that server. It might be necessary to transfer new containers
to the server that is taking over.  

View raw message