helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Greg Brandt <brandt.g...@gmail.com>
Subject Re: Helix IPC update
Date Mon, 11 Aug 2014 20:13:10 GMT
Sure, here is the repo: https://github.com/brandtg/helix-actors, the bulk
of implementation:
https://github.com/brandtg/helix-actors/blob/master/helix-actors/src/main/java/org/apache/helix/ipc/netty/NettyHelixIPCService.java,
and the benchmark code:
https://github.com/brandtg/helix-actors/blob/master/helix-actors/src/test/java/org/apache/helix/ipc/benchmark/BenchmarkDriver.java

The experiment consisted of running BenchmarkDriver on two machines
connected by a 1 GigE switch. BenchmarkDriver was run for 8 partitions
using 1 thread to generate traffic, consisting of ~1KB messages. It
continually sends messages to a target instance, and sends acknowledgement
messages to the sender for each message it receives. Both machines were
configured to send traffic to each other.

Adding another method that takes byte[] is a good idea. We can use
io.netty.buffer.Unpooled#wrappedBuffer to avoid data copy as well.

We don't necessarily need a UUID for message ID; 16B have just been
reserved for the message ID portion, and users can always use UUID(long
mostSigBits, long leastSigBits) to avoid potentially costly computation of
random UUIDs.

The message structure is this:

     +----------------------+
     | totalLength (4B)     |
     +----------------------+
     | version (4B)         |
     +----------------------+
     | messageType (4B)     |
     +----------------------+
     | messageId (16B)      |
     +----------------------+
     | len | cluster        |
     +----------------------+
     | len | resource       |
     +----------------------+
     | len | partition      |
     +----------------------+
     | len | state          |
     +----------------------+
     | len | srcInstance    |
     +----------------------+
     | len | dstInstance    |
     +----------------------+
     | len | message        |
     +----------------------+

Where len is 4B integer specifying the length of the subsequent piece of
cluster metadata. So for example, a message from "localhost_8000" to
"localhost_9000" in cluster "STORAGE" for resource "MyDB" to the "MASTER"
of "MyDB_0" would have: 4 + 4 + 4 + 16 + (4 + 7) + (4 + 4) + (4 + 6) + (4 +
6) + (4 + 14) + (4 + 14) + 4 = 107B of overhead. Not too bad when you
consider common use cases: control events, which may be small, but
infrequent; and data replication, which may be frequent, but overhead
easily amortized by size of data.

-Greg


On Mon, Aug 11, 2014 at 11:45 AM, kishore g <g.kishore@gmail.com> wrote:

> Hi Greg,
>
> This is great, do you have the code some where that I can take a look. Can
> you give some description on the experiment you ran to test the performance.
>
> ByteBuf is definitely valuable but in most cases user constructs the
> ByteBuf after converting the message object into byte[]. We can probably
> add another method that takes in a byte[] and we create a ByteBuf out of
> it. I would still keep the method that takes in ByteBuf but keep it for
> advanced users.
>
> Is UUID too heavy for messageId, what if its just an integer or logical
> sequence id that is monotonically incremented by the sender.
>
> Also has the payload structure changed, what is the overhead per message.
>
> thanks,
> Kishore G
>
>
>
> On Mon, Aug 11, 2014 at 10:33 AM, Greg Brandt <brandt.greg@gmail.com>
> wrote:
>
>> We've fleshed out the idea for Helix IPC a little more (HELIX-470), and
>> results
>> are fairly promising.
>>
>> The prototype is Netty-based, and our benchmark was able to generate
>> ~1Gbps
>> traffic (on network w/ 1 GigE switch), where message sizes were ~1KB.
>>
>> The API is simplified somewhat from the original ticket. The idea is that
>> we
>> just provide the most basic transport layer possible to allow for maximum
>> flexibility at the application layer.
>>
>> One can send messages and register callbacks using a HelixIPCService:
>>
>> public interface HelixIPCService {
>>     void send(Set<HelixAddress> destinations, int messageType, UUID
>> messageId, ByteBuf message);
>>     void registerCallback(int messageType, HelixIPCCallback callback);
>> }
>>
>> public interface HelixIPCCallback {
>>     void onMessage(HelixMessageScope scope, UUID messageId, ByteBuf
>> message);
>> }
>>
>> A HelixResolver is used to generate the HelixAddress(es) that correspond
>> to a
>> HelixMessageScope (i.e. physical machine addresses):
>>
>> public interface HelixResolver {
>>     Set<HelixAddress> getDestinations(HelixMessageScope scope);
>>     HelixAddress getSource(HelixMessageScope scope);
>> }
>>
>> And a HelixMessageScope contains the cluster, resource, partition, state,
>> and
>> sender information that fully describe the message's path in the cluster.
>>
>> One thing to note is that Netty's ByteBuf is part of the API here. This is
>> chosen specifically to avoid memory copies when doing things like data
>> replication. Also, it's available via io.netty:netty-buffer, so this
>> doesn't
>> mean one needs to pull in all Netty dependencies. This is preferable to
>> java.nio.ByteBuffer because it has support for buffer pooling and a much
>> richer
>> API.
>>
>> -Greg
>>
>>
>

Mime
View raw message