incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthieu Morel <mmo...@apache.org>
Subject Re: S4 Load Balancing Mechanism
Date Thu, 11 Oct 2012 11:00:22 GMT
On 10/11/12 12:35 PM, Daniel Gómez Ferro wrote:
>
>
> On Thu Oct 11 05:52:01 2012, Frank Zheng wrote:
>> Hi All,
>>
>> I want to figure out the mechanism of load balancing in S4.
>> In the paper and web site, it says ZooKeeper takes this
>> responsibility, but the description is not very clear.
>> I want to figure out that how ZooKeeper does load balancing.
>> For example, if there are two working PE nodes, and 120 messages per
>> second. The adapter distributes these messages evenly, 60 per PE node,
>> right?
>> When one PE node is added into the cluster, does it mean ZooKeeper
>> balances the messages, 40 per PE node?
>> And how does ZooKeeper decide which message to be distributed to which
>> node?
>
> Hi,
>
> Currently load balancing is static. When you define your cluster you
> specify the number of tasks (say, 2) which won't change for this
> cluster. If you have more than 2 nodes, the rest will be backup nodes.
> In case one of the serving nodes fails, a backup node will pick up the
> orphaned task and start serving requests, but the load will always be
> spread among the 2 tasks that you defined at the beginning.
>
> Having dynamic load balancing would be really cool, but I don't know if
> there are currently any efforts in that direction.

There is the zeno prototype https://github.com/s4/zeno , which is 
actually independent of S4. It also relies on ZooKeeper for metadata 
about partitions.

Matthieu

Mime
View raw message