helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Greg Brandt <brandt.g...@gmail.com>
Subject Add performant IPC (Helix actors)
Date Fri, 11 Jul 2014 17:56:43 GMT
(copied from HELIX-470)

Helix is missing a high-performance way to exchange messages among resource
partitions, with a user-friendly API.

Currently, the Helix messaging service relies on creating many nodes in
ZooKeeper, which can lead to ZooKeeper outages if messages are sent too

In order to avoid this, high-performance NIO-based HelixActors should be
implemented (in rough accordance with the actor model). HelixActors exchange
messages asynchronously without waiting for a response, and are

The API would look something like this:

public interface HelixActor<T> {
    void send(Partition partition, String state, T message);
    void register(String resource, HelixActorCallback<T> callback);
public interface HelixActorCallback<T> {
    void onMessage(Partition partition, State state, T message);

#send should likely support wildcards for partition number and state, or
its method signature might need to be massaged a little bit for more
flexibility. But that's the basic idea.

Nothing is inferred about the format of the messages - the only metadata we
need to be able to interpret is (1) partition name and (2) state. The user
provides a codec to encode / decode messages, so it's nicer to implement
HelixActor#send and HelixActorCallback#onMessage.

public interface HelixActorMessageCodec<T> {
    byte[] encode(T message);
    T decode(byte[] message);

Actors should support somewhere around 100k to 1M messages per second. The
Netty framework is a potential implementation candidate, but should be
thoroughly evaluated w.r.t. performance.

View raw message