incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthieu Morel <mmo...@apache.org>
Subject Re: questions about dynamic load balancing and nodes on multiple machines
Date Fri, 24 Feb 2012 19:00:18 GMT
On 2/24/12 5:28 PM, Karthik Kambatla wrote:
> Matthieu,
>
> Good to know about the prototype - can you point us to any document
> explaining the design adopted for load-balancing?

Unfortunately I am not aware of any public documentation for that 
project. The idea though, as far as I understand it, is that it uses use 
medium-grained partitions of PEs that it monitors and swaps around for 
homogenizing usage of resources.

Matthieu

>
> Thanks
> Karthik
>
> On Fri, Feb 24, 2012 at 8:15 AM, Matthieu Morel <mmorel@apache.org
> <mailto:mmorel@apache.org>> wrote:
>
>     On 2/24/12 12:27 PM, Jia Ding wrote:
>
>         Hi everyone,
>
>         these days I came across S4, and the document is very helpful.
>         But I'm not so clear about the dynamic load-balancing of S4, and
>         Processing Nodes(PN), Processing Element(PE) running on multiple
>         machines.
>
>         I have some questions about these:
>
>         1) Can one PN run across multiple machines? e.g. one PN has 10
>         PEs, is
>         is possible
>         to run 5 PEs in machine A and other 5 PEs in machine B?
>
>
>     The deployment is currently symmetrical: all prototypes are deployed
>     on all nodes, but PNs hold different keyed PE instances (according
>     to the keys and partitioning scheme)
>
>
>
>         2) If the question 1's situation is possible, when machine A is
>         busy,
>         machine B is relative free,
>         can migrate a PE from machine A to machine B for dynamic
>         load-balancing?
>
>         3) In one PN, when a PE is overload, what S4 will do? Does S4
>         automaticlly create another
>         instance of this PE?
>
>
>         4) I've read the document about Zookeeper, and I know that can
>         run S4 in
>         dymanic mode.
>         And I also know that S4 can run with multiple nodes and multiple
>         machines.
>         e.g. there are 3 PNs running on machine A and 1 PNs running on
>         machine B.
>         When machine A is busy, can man dynamic migrate a PN from
>         machine A to
>         machine B
>         for the load-balancing.
>
>         5) There is one PN in machine A. machine B has no node.
>         When this PN is too busy, can it be cloned and put its clone to
>         machine B?
>
>
>     Load balancing is not available at the moment in the open source
>     release, but there is a prototype available here
>     https://github.com/s4/zeno
>
>     Regards,
>
>     Matthieu
>
>


Mime
View raw message