apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Munagala Ramanath <...@datatorrent.com>
Subject Re: Apex Communication Protocols
Date Fri, 25 Nov 2016 19:27:22 GMT
A brief description is here:
http://docs.datatorrent.com/beginner/#buffer-server

It is indeed part of apex-core; see:
https://github.com/apache/apex-core/tree/master/engine/src/main/java/com/datatorrent/stram/
as well as the *stream* subdirectory.

Ram

On Fri, Nov 25, 2016 at 11:13 AM, Max Bridgewater <max.bridgewater@gmail.com
> wrote:

> Awesome. Thanks for the additional details. I can't find any reference to
> Buffer Server online. Could you give some pointers? or is it part of the
> Apex core?
>
> Regards,
> Max.
>
> On Fri, Nov 25, 2016 at 1:57 PM, Ashwin Chandra Putta <
> ashwinchandrap@gmail.com> wrote:
>
>> Max,
>>
>> Apex deploys a fast pub-sub server called buffer server in every yarn
>> container it gets (except AM) before deploying operators on it. For all the
>> operators which are connected downstream to operators outside the current
>> container, their output ports become publishers to the buffer server. The
>> downstream operators' input ports become subscribers to the buffer server.
>> So there is no concept a central operator/port registry, however all the
>> downstream operators do register their input ports with the buffer server.
>> The serialization algorithm is kryo based. The data transport protocol is
>> based on Netlet: https://github.com/DataTorrent/Netlet which is on top
>> of TCP/IP.
>>
>> Regards,
>> Ashwin.
>>
>> On Fri, Nov 25, 2016 at 8:33 AM, Max Bridgewater <
>> max.bridgewater@gmail.com> wrote:
>>
>>> Hi Folks,
>>>
>>> I was giving an Apex demo the other day and people asked following
>>> questions:
>>>
>>> 1) what is the communication protocol between operators when they are on
>>> distant nodes. That means, how does Apex transport the tuples from one node
>>> to the other?
>>> Is it a custom protocol on top of TCP/IP or is it RPC?
>>> 2) What is the serialization algorithm used?
>>> 3) What is the addressing scheme between operators? That means how does
>>> Apex know where an operator is located and how to route data to it? Is
>>> there an operator registry? If so, where does it reside?
>>>
>>> Thoughts?
>>>
>>> Thanks,
>>> Max.
>>>
>>>
>>
>>
>> --
>>
>> Regards,
>> Ashwin.
>>
>
>

Mime
View raw message