helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "vlad.gm@gmail.com" <vlad...@gmail.com>
Subject Re: Add performant IPC (Helix actors)
Date Fri, 11 Jul 2014 18:07:03 GMT
This sounds like service discovery for a messaging solution. At this point
there would be some overlap with message buses such as Finagle. Perhaps
this should be a general framework for discovering the service endpoint
that could leave the actual underlaying messaging stack open (be it
Finagle, Netty-based or ZeroMQ, for example). My only fear is that if the
messaging framework is encapsulated completely into Helix, it would be hard
to compete with tuned messaging bus solutions and cover all possible use
cases (for example, in my company, we use a large number of sub cases on
the synchronous call to asynchronous call spectrum).


On Fri, Jul 11, 2014 at 10:56 AM, Greg Brandt <brandt.greg@gmail.com> wrote:

> (copied from HELIX-470)
> Helix is missing a high-performance way to exchange messages among
> resource partitions, with a user-friendly API.
> Currently, the Helix messaging service relies on creating many nodes in
> ZooKeeper, which can lead to ZooKeeper outages if messages are sent too
> frequently.
> In order to avoid this, high-performance NIO-based HelixActors should be
> implemented (in rough accordance with the actor model). HelixActors exchange
> messages asynchronously without waiting for a response, and are
> partition/state-addressable.
> The API would look something like this:
> public interface HelixActor<T> {
>     void send(Partition partition, String state, T message);
>     void register(String resource, HelixActorCallback<T> callback);
> }
> public interface HelixActorCallback<T> {
>     void onMessage(Partition partition, State state, T message);
> }
> #send should likely support wildcards for partition number and state, or
> its method signature might need to be massaged a little bit for more
> flexibility. But that's the basic idea.
> Nothing is inferred about the format of the messages - the only metadata
> we need to be able to interpret is (1) partition name and (2) state. The
> user provides a codec to encode / decode messages, so it's nicer to
> implementHelixActor#send and HelixActorCallback#onMessage.
> public interface HelixActorMessageCodec<T> {
>     byte[] encode(T message);
>     T decode(byte[] message);
> }
> Actors should support somewhere around 100k to 1M messages per second. The
> Netty framework is a potential implementation candidate, but should be
> thoroughly evaluated w.r.t. performance.

View raw message