apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vlad Rozov <v.ro...@datatorrent.com>
Subject Re: Using queue for publisher to buffer server communication (APEX-259)
Date Mon, 16 Nov 2015 03:57:52 GMT
JDK ArrayBlockingQueue is the slowest BlockingQueue implementation. On 
my machine passing 100000000 objects from one thread to another takes 28 
seconds using ArrayBlockingQueue, 16 seconds using CircularBuffer and 
0.1 seconds using SpscArrayQueue. In all cases queue capacity was around 
65000 elements. I guess we should be able to pass data 500 times faster 
compared to loopback interface.

Thank you,

Vlad

On 11/14/15 14:40, Munagala Ramanath wrote:
> I was curious about this so I wrote a couple of small standalone
> programs, one using the loopback
> interface and one using ArrayBlockingQueue to send byte arrays from
> one thread to another.
>
> The loopback version is about 50% slower than the ArrayBlockingQueue:
>
> Reader: read 4000000 tuples of size 128 in 3893 ms    (ArrayBlockingQueue)
> Writer: wrote 4000000 tuples of size 128 in 3893 ms
>
> Writer: wrote 4000000 tuples of size 128 in 5783 ms    (Loopback)
> Reader: read 4000000 tuples of size 128 in 5820 ms
>
> Ram
>
> On Fri, Nov 13, 2015 at 11:19 AM, Isha Arkatkar <isha@datatorrent.com> wrote:
>> Hi all,
>>
>>     For APEX-259 (https://malhar.atlassian.net/browse/APEX-259), I am
>> exploring option of passing serialized tuples from publisher to buffer
>> server through a blocking queue.
>>
>> Right now, publisher and buffer server reside within the same container,
>> however, communication between the two goes though sockets. We want to
>> check if we get any performance benefit by changing this communication to
>> queue-based one.
>>
>> This is in exploration phase right now, but if we do see improvement, we
>> may want to provide it as a pluggable option.
>>
>> Please let me know your thoughts!
>>
>> Thanks,
>> Isha


Mime
View raw message