curator-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Vines <>
Subject Re: Multiple consumers on a single server - strange behavior.
Date Sun, 17 Nov 2013 17:23:10 GMT
I don't know the finer details, but I'm not surprised by your results.
Having 15 consumers of a single, highly synchronous queue is hard to
maintain, so it makes sense that each consumer would spend more time
idling. I think ultimately you want to scale the number of consumers to the
complexity of the processing that gets done after an item is removed from
the queue, not arbitrarily scaling up.

Sent from my phone, please pardon the typos and brevity.
On Nov 17, 2013 10:54 AM, "Sznajder ForMailingList" <> wrote:

> If someone has hint on this subject, it could be very useful for me...
> Thanks
> Benjamin
> On Thu, Nov 14, 2013 at 11:18 PM, Sznajder ForMailingList <
>> wrote:
>> Hi
>> I made a short test as following:
>> - I have a chorum of 3 nodes for Zookeeper.
>> - I wrote a class using Curator QueueProducer who produces all the time
>> (when the queue is 10% full, it creates new items) , items (random integer)
>> - I wrote a simple class using Curator Queue Consumer which simply prints
>> to Log "consumed item i".
>> I tested some different combinations :
>> - running the consumers on one, two or three nodes.
>> - running one or more consumers in parallel on a given node.
>> But, and here is my question: I see some very strange behavior when I
>> have several consummers in parallel on a node. For example, running 5
>> consumers per node on 3 nodes, I see a throughput **very** slow. When
>> looking at my Log, I see that most of the consumers are most of the time on
>> an idle state....
>> Do I mistake somewhere?
>>  I was expecting to enhance the throughput by augmenting the number of
>> consumers, I am surprised to see the opposite....
>> Thanks a lot
>> Benjamin

View raw message