incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthieu Morel <>
Subject Re: Decoupling partitions and nodes
Date Fri, 11 Nov 2011 14:05:23 GMT
On 11/11/11 6:11 AM, Karthik Kambatla wrote:
> Hi
> In last Tuesday's meeting, we agreed on decoupling partitions and nodes.
> How should we go about it?
>     1. io.s4.comm.topology.ClusterNode currently has a field called
>     partition - it should be changed to a set of partitions.
>     2. Emitters need to maintain hashtables<partition, clusterNode>; send
>     will pick the right node from this table.
>     3. Listeners need to receive messages for each partition that they are
>     hosting.
> Comments?
> Thanks
> Karthik
> PS: The NettyEmitter code that I am trying to fix depends on how we decide
> to resolve this.
Hi Karthik,

My understanding of ClusterNode is that it corresponds to a task that 
was picked by a process: it maps a logical partition to a physical node. 
The ensemble of ClusterNodes is reflected in the Topology, and updated 
automatically upon any change (at least, for the Zookeeper version).

Therefore I wouldn't do anything about point 1.

I agree though that the documentation doesn't say much.

I added a few improvements to the comm layer in my latest commit 
yesterday, that enables integration with Zookeeper. Maybe you could have 
a look at that and see what else would be missing? Points 2 and 3 are 
already implemented, as I see it.



View raw message