curator-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sznajder ForMailingList <bs4mailingl...@gmail.com>
Subject Re: Multiples queue consumers in a quorum reduces the throughput
Date Thu, 28 Nov 2013 11:56:20 GMT
Hi Jordan,

Thanks a lot for your help!

In the meanwhile, I think I found the problem. I open another thread for
this issue.

Best regards and thanks again.


Benjamin


On Tue, Nov 26, 2013 at 2:13 AM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:

> I’m not sure how I can help on this. ZooKeeper is not a good datastore
> system. ZooKeeper is a distributed coordination system. Each queue consumer
> must call getChildren() for the message path and attempt to process every
> message in the path. There’s no way around this given ZooKeeper’s current
> APIs. This is why the Curator docs say that “ZooKeeper is a very bad Queue”:
>
> https://cwiki.apache.org/confluence/display/CURATOR/TN4
>
> -Jordan
>
> On Nov 24, 2013, at 8:51 AM, Sznajder ForMailingList <
> bs4mailinglist@gmail.com> wrote:
>
> Hi
>
> A parallel problem to the problem I described earlier in this list is a
> following:
>
> The parameters are still the same:
>
> - I have a producer who produces items in the queue (integer). The
> producer adds some items when the queue is half of its initial size.
>
> - The consumer just picks the item and write to a log file and goes to
> sleep a while...
>
>
> When I define a chorum of 2 machines - One producer, and one consumer, I
> consume 1320 items in 5 minutes.
>
> When I define a quorum of 3 machines - One producer and two consumers, I
> consume 2693 items in 5 minutes (expected!)
>
> When I define a quorum of 4 machines - One producer and three consumers, I
> suddenly, consume only 1975 items in 5 minutes!! (Even less than the
> precedent configuration!!)
>
> I would thank you for your help/. This issue is relatively urgent for me..
>
> Best regards
> Benjamin
> <MyQueueConsumer.java><QueueProducer.java>
>
>
>

Mime
View raw message