zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sznajder ForMailingList <bs4mailingl...@gmail.com>
Subject Multi threaded queue consumers
Date Sun, 24 Nov 2013 16:27:26 GMT
Hi,

I share with you a very problematic (for me...) issue, I encountered.

I wrote a very simple Queue (Producer and consumer).

The producer, put to the ZK Queue items (integer) when the queue reached
half of its initial size.
And Consumer just picks from the list and prints to a log file the consumed
item and sleeps some time.

In addition, I defined the Consumer to be multi-threaded

I tested the following configurations on different quorums (when each
server is 16 cores)

 Quorum of 2 servers : One consumer and one producer, 2 threads for the
consumer:
In 5 minutes, I consumed 1320 items.

When I make the consumer to be 16 threads, I get: 10890 items consumed. All
is OK.




However, when I move to a quorum of three, with 2 consumers (and each
consumer 16 threads), I suddenly get:
In 5 minutes 12634 items consumed. That means, that adding a new consumer
did not enhance the throughput of the system.


I am blocked with this issue, several days, and any help will be welcome!!

Benjamin

Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message