helix-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kanak Biscuitwala (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HELIX-128) Support multiple communication channels between controllers, participants, and spectators
Date Thu, 13 Feb 2014 07:20:23 GMT

     [ https://issues.apache.org/jira/browse/HELIX-128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Kanak Biscuitwala updated HELIX-128:
------------------------------------

    Description: 
It's clearly important for members of a cluster to communicate with one another.

Currently all communications between controller, participant, and spectator happens via zookeeper.
This is important for systems where fault tolerance cannot be compromised and consensus is
necessary. But this also increases the latency of communication quite a bit. 

For some applications, it is important that communication between different components is
fast, it's ok to lose messages in some cases or if controller fails after the message is sent.
we can either use tcp or any other pub/sub mechanism to make this as faster.

Furthermore, members of the cluster may want to transfer larger amounts of data while still
routing based on current Helix assignments. One way to do this is to wrap the current messaging
API, but use a different underlying framework. Netty provides an efficient framework for intra-cluster
communication and may be a good fit here.

The big thing is that we don't want ZooKeeper to turn into a data store, nor do we want it
to be a place where there is high write traffic of any size. This is why we need an alternative
channel.

  was:
It's clearly important for members of a cluster to communicate with one another.

Currently all communications between controller, participant, and spectator happens via zookeeper.
This is important for systems where fault tolerance cannot be compromised and consensus is
necessary. But this also increases the latency of communication quite a bit. 

For some applications, it is important that communication between different components is
fast, its ok to lose messages in some cases or if controller fails after the message is sent.
we can either use tcp or any other pub/sub mechanism to make this as faster.

Furthermore, members of the cluster may want to transfer larger amounts of data while still
routing based on current Helix assignments. One way to do this is to wrap the current messaging
API, but use a different underlying framework. Netty provides an efficient framework for intra-cluster
communication and may be a good fit here.

The big thing is that we don't want ZooKeeper to turn into a data store, nor do we want it
to be a place where there is high write traffic of any size. This is why we need an alternative
channel.


> Support multiple communication channels between controllers, participants, and spectators
> -----------------------------------------------------------------------------------------
>
>                 Key: HELIX-128
>                 URL: https://issues.apache.org/jira/browse/HELIX-128
>             Project: Apache Helix
>          Issue Type: Improvement
>            Reporter: Kanak Biscuitwala
>
> It's clearly important for members of a cluster to communicate with one another.
> Currently all communications between controller, participant, and spectator happens via
zookeeper. This is important for systems where fault tolerance cannot be compromised and consensus
is necessary. But this also increases the latency of communication quite a bit. 
> For some applications, it is important that communication between different components
is fast, it's ok to lose messages in some cases or if controller fails after the message is
sent. we can either use tcp or any other pub/sub mechanism to make this as faster.
> Furthermore, members of the cluster may want to transfer larger amounts of data while
still routing based on current Helix assignments. One way to do this is to wrap the current
messaging API, but use a different underlying framework. Netty provides an efficient framework
for intra-cluster communication and may be a good fit here.
> The big thing is that we don't want ZooKeeper to turn into a data store, nor do we want
it to be a place where there is high write traffic of any size. This is why we need an alternative
channel.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message