incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore g <>
Subject Re: S4 Communication Layer using ZooKeeper
Date Wed, 03 Oct 2012 04:51:53 GMT
At a very high level, this is how cluster management works

Each s4 cluster has a name space reserved /clustername in zookeeper. There
is an initial setup process where one or many znodes are created under
/clustername/tasks. When nodes join the cluster they check if some one has
already claimed a task by looking at /clustername/process/, if not it grabs
the lock by creating an ephemeral node under /clustername/process/. If all
tasks are taken it becomes a standby node. When any active node dies, the
standby node gets notified and tries to grab the lock.

We can provide more details, if you can let us know which aspect of cluster
management mechanism you are interested in.

Kishore G

On Tue, Oct 2, 2012 at 9:17 PM, Frank Zheng <> wrote:

> Hi All,
> I am exploring the cluster management mechanism and fault tolerance of S4.
> I saw that S4 used ZooKeeper in the communication layer. But it seems not
> very clear in that pater, " S4: Distributed Stream Computing Platform".
> I tried to search the reference "[15] Communication layer using ZooKeeper,
> Yahoo! Inc. Tech. Rep., 2009", but it is not available.
> Could anyone introduce me the role of ZooKeeper in S4, and the cluster
> management mechanism in detail?
> Thanks.
> Sincerely,
> Frank

View raw message