Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
Received-SPF: pass (athena.apache.org: domain of devaki.vamsi@gmail.com
 designates 209.85.217.170 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAM_4j5dwQSx0NPxf_i9nU0a0tQnyV89mMZMH1ZdUBLsyOMgwEQ@mail.gmail.com>
References: 
 <CAM_4j5dwQSx0NPxf_i9nU0a0tQnyV89mMZMH1ZdUBLsyOMgwEQ@mail.gmail.com>
From: Vamsi Devaki <devaki.vamsi@gmail.com>
Date: Mon, 25 Nov 2013 12:13:14 +0530
Message-ID: 
 <CAFuMMGQ3qDzgrUKa2prR81AQ7pkc3MZr_EnTRuubJKUQonOr1A@mail.gmail.com>
Subject: Re: Multi threaded queue consumers
To: user@zookeeper.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Sun, Nov 24, 2013 at 9:57 PM, Sznajder ForMailingList
<bs4mailinglist@gmail.com> wrote:
> Hi,
>
> I share with you a very problematic (for me...) issue, I encountered.
>
> I wrote a very simple Queue (Producer and consumer).
>
> The producer, put to the ZK Queue items (integer) when the queue reached
> half of its initial size.
> And Consumer just picks from the list and prints to a log file the consumed
> item and sleeps some time.
>
> In addition, I defined the Consumer to be multi-threaded
>
> I tested the following configurations on different quorums (when each server
> is 16 cores)
>
>  Quorum of 2 servers : One consumer and one producer, 2 threads for the
> consumer:
> In 5 minutes, I consumed 1320 items.
>
> When I make the consumer to be 16 threads, I get: 10890 items consumed. All
> is OK.
>
>
>
>
> However, when I move to a quorum of three, with 2 consumers (and each
> consumer 16 threads), I suddenly get:
> In 5 minutes 12634 items consumed. That means, that adding a new consumer
> did not enhance the throughput of the system.
>
>
> I am blocked with this issue, several days, and any help will be welcome!!
>
> Benjamin
>

Hello Benjamin,

Did you measure add/delete thru-put for your cluster? If not, can you
check if it is significantly higher than 12k?

This should help to get you started:
https://github.com/brownsys/zookeeper-benchmark

Regards,
Vamsi