Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
Received-SPF: pass (athena.apache.org: domain of bs4mailinglist@gmail.com
 designates 209.85.215.180 as permitted sender)
MIME-Version: 1.0
Date: Sun, 24 Nov 2013 18:27:26 +0200
Message-ID: 
 <CAM_4j5dwQSx0NPxf_i9nU0a0tQnyV89mMZMH1ZdUBLsyOMgwEQ@mail.gmail.com>
Subject: Multi threaded queue consumers
From: Sznajder ForMailingList <bs4mailinglist@gmail.com>
To: user@zookeeper.apache.org
Content-Type: multipart/mixed; boundary=047d7b343cfe1052d904ebeeb971

--047d7b343cfe1052d904ebeeb971
Content-Type: multipart/alternative; boundary=047d7b343cfe1052d604ebeeb96f

--047d7b343cfe1052d604ebeeb96f
Content-Type: text/plain; charset=ISO-8859-1

Hi,

I share with you a very problematic (for me...) issue, I encountered.

I wrote a very simple Queue (Producer and consumer).

The producer, put to the ZK Queue items (integer) when the queue reached
half of its initial size.
And Consumer just picks from the list and prints to a log file the consumed
item and sleeps some time.

In addition, I defined the Consumer to be multi-threaded

I tested the following configurations on different quorums (when each
server is 16 cores)

 Quorum of 2 servers : One consumer and one producer, 2 threads for the
consumer:
In 5 minutes, I consumed 1320 items.

When I make the consumer to be 16 threads, I get: 10890 items consumed. All
is OK.


However, when I move to a quorum of three, with 2 consumers (and each
consumer 16 threads), I suddenly get:
In 5 minutes 12634 items consumed. That means, that adding a new consumer
did not enhance the throughput of the system.


I am blocked with this issue, several days, and any help will be welcome!!

Benjamin

--047d7b343cfe1052d604ebeeb96f
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div>Hi,<br><br>I share with you a very problema=
tic (for me...) issue, I encountered.<br><br></div>I wrote a very simple Qu=
eue (Producer and consumer).<br><br>The producer, put to the ZK Queue items=
 (integer) when the queue reached half of its initial size.<br>
</div>And Consumer just picks from the list and prints to a log file the co=
nsumed item and sleeps some time.<br><br></div><div>In addition, I defined =
the Consumer to be multi-threaded<br></div><div><br></div><div>I tested the=
 following configurations on different quorums (when each server is 16 core=
s)<br>
<br></div><div>=A0Quorum of 2 servers : One consumer and one producer, 2 th=
reads for the consumer:<br></div><div>In 5 minutes, I consumed 1320 items.<=
br><br></div><div>When I make the consumer to be 16 threads, I get: 10890 i=
tems consumed. All is OK.<br>
<br><br><br><br></div><div>However, when I move to a quorum of three, with =
2 consumers (and each consumer 16 threads), I suddenly get:<br></div><div>I=
n 5 minutes 12634 items consumed. That means, that adding a new consumer di=
d not enhance the throughput of the system.<br>
<br><br></div><div>I am blocked with this issue, several days, and any help=
 will be welcome!!<br><br>Benjamin<br></div><div><br><br><br></div><br></di=
v>

--047d7b343cfe1052d604ebeeb96f--

--047d7b343cfe1052d904ebeeb971--