curator-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jordan Zimmerman <jor...@jordanzimmerman.com>
Subject Re: Multiples queue consumers in a quorum reduces the throughput
Date Tue, 26 Nov 2013 00:13:04 GMT
I’m not sure how I can help on this. ZooKeeper is not a good datastore system. ZooKeeper
is a distributed coordination system. Each queue consumer must call getChildren() for the
message path and attempt to process every message in the path. There’s no way around this
given ZooKeeper’s current APIs. This is why the Curator docs say that “ZooKeeper is a
very bad Queue”:

https://cwiki.apache.org/confluence/display/CURATOR/TN4

-Jordan

On Nov 24, 2013, at 8:51 AM, Sznajder ForMailingList <bs4mailinglist@gmail.com> wrote:

> Hi
> 
> A parallel problem to the problem I described earlier in this list is a following:
> 
> The parameters are still the same:
> 
> - I have a producer who produces items in the queue (integer). The producer adds some
items when the queue is half of its initial size.
> 
> - The consumer just picks the item and write to a log file and goes to sleep a while...
> 
> 
> When I define a chorum of 2 machines - One producer, and one consumer, I consume 1320
items in 5 minutes.
> 
> When I define a quorum of 3 machines - One producer and two consumers, I consume 2693
items in 5 minutes (expected!)
> 
> When I define a quorum of 4 machines - One producer and three consumers, I suddenly,
consume only 1975 items in 5 minutes!! (Even less than the precedent configuration!!)
> 
> I would thank you for your help/. This issue is relatively urgent for me..
> 
> Best regards
> Benjamin
> <MyQueueConsumer.java><QueueProducer.java>


Mime
View raw message