From users-return-27388-apmail-activemq-users-archive=activemq.apache.org@activemq.apache.org Wed May 4 20:30:17 2011
Return-Path:
X-Original-To: apmail-activemq-users-archive@www.apache.org
Delivered-To: apmail-activemq-users-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
by minotaur.apache.org (Postfix) with SMTP id 906073A6E
for ; Wed, 4 May 2011 20:30:17 +0000 (UTC)
Received: (qmail 80711 invoked by uid 500); 4 May 2011 20:30:17 -0000
Delivered-To: apmail-activemq-users-archive@activemq.apache.org
Received: (qmail 80675 invoked by uid 500); 4 May 2011 20:30:17 -0000
Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
List-Help:
List-Unsubscribe:
List-Post:
List-Id:
Reply-To: users@activemq.apache.org
Delivered-To: mailing list users@activemq.apache.org
Received: (qmail 80667 invoked by uid 99); 4 May 2011 20:30:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 20:30:17 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
tests=SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jcarlson@e-dialog.com designates 208.94.20.29 as permitted sender)
Received: from [208.94.20.29] (HELO corp-mail.e-dialog.com) (208.94.20.29)
by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 20:30:11 +0000
Received: from eeyore.ad.e-dialog.com ([::1]) by eeyore.ad.e-dialog.com
([::1]) with mapi; Wed, 4 May 2011 16:29:49 -0400
From: Josh Carlson
To: "users@activemq.apache.org"
Date: Wed, 4 May 2011 16:29:48 -0400
Subject: RE: Scaleability problems with queue subscriptors
Thread-Topic: Scaleability problems with queue subscriptors
Thread-Index: AcwGvLBMl3Fj7jjkRom5NDVhcoRSNQD2yNKg
Message-ID: <3A2A0F51850C264790C6008B7F92D0470BBAFBDEC7@eeyore.ad.e-dialog.com>
References: <3A2A0F51850C264790C6008B7F92D0470BBAFBD8C5@eeyore.ad.e-dialog.com>
In-Reply-To:
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org
Hi Gary,
Thanks for the response. We've decided it would be easy for us to partition=
our consumers such that they each consumer operates on only one queue. How=
ever, the model we are using retrieves a message from one queue (the job qu=
eue), then grabs something to do from another queue (the work queue), once =
it retrieves the message from the work queue it acknowledges the job queue =
and goes and does its work. However, since another message is dispatched on=
ce the ack is done and the work can take a long time (potentially infinite)=
we unsubscribe. Subsequently, once the work is done the consumer needs to=
go subscribe and retrieve another message.
Switching to one queue helps when there is no or few messages. However, it =
is not scaling when there are plenty of messages due to the way we need to =
subscribe/unsubscribe. Do you have any suggestions on how we might be able =
to scale this given our current architecture?
-Josh
> -----Original Message-----
> From: Gary Tully [mailto:gary.tully@gmail.com]
> Sent: Friday, April 29, 2011 6:27 PM
> To: users@activemq.apache.org
> Subject: Re: Scaleability problems with queue subscriptors
>=20
> setting up a consumer is a little expensive, have a look at using a
> composite destination so that you can subscribe to all destinations at
> once.
> Also, there is a delay between new consumer registration and async
> dispatch, so waiting a few seconds before unsubscribe is necessary.
>=20
> http://activemq.apache.org/composite-destinations.html
>=20
> On 28 April 2011 23:41, Josh Carlson wrote:
> > We are using a shared file system Master/Slave for the broker. Version
> 5.4.2. Our clients use the STOMP protocol. We use client
> acknowledgements and communicate synchronously with the broker (using
> receipts). We set prefetch to 1 in our subscriptions. Our clients
> iterate over several queues, subscribing, checking for messages (timeout
> of 50ms), and if one isn't available it un-subscribes and goes to the
> next queue. There are almost always cases where there are no messages in
> the queues. We ran into a problem where our application slowed down to a
> crawl when we deployed additional clients and I've narrowed it down to
> the fact that most of the time when we subscribed to the queue and then
> asked if a message was ready it wouldn't be even though there were
> messages in the queue. My assumption is that it is taking some time to
> dispatch the message.
> >
> > Is there some configuration parameters I might want to set to help
> with this problem? Or is this type of use just not going to scale?
> >
> > Here is some benchmark data. Each test, creates N consumers, but
> before they are allowed to start it enqueues 50*N messages for the
> consumer into *one* queue. The first set of metrics is for the case
> where the consumers are iterating over 6 different queues (even though
> there is only data in one). The second set of metrics we ONLY have 1
> queue ... in this case the client only subscribes and un-subscribes once
> except in the case where a message 'isn't ready' in that 50ms (in which
> case it re-subscribes to the same queue). The metrics capture the entire
> process. getNextMessage, iterates over the queues, doing the
> subscribes/un-subscribes, receipts etc ...
> >
> > Note that in the 6 queue case time degrades once you have 100
> consumers. In the other case it degrades after 100 but we never see a
> Median greater than 206ms.
> >
> > TEST Case 6 Queues ... 5 of which are empty (note that in this first
> case since 5 queues are empty one expects at least 250ms to poll those 5
> empty queues). Times are in seconds.
> >
> > Number of Consumers 1. Muliple Queues
> > Min: 0.349334999918938
> > Max: 0.368788999971002
> > Mean: 0.350222800001502
> > Median: 0.349644500005525
> > Std Dev: 0.00271797410606451
> > Starting test for consumer count 10
> >
> > Number of Consumers 10. Muliple Queues
> > Min: 0.349282000097446
> > Max: 0.394184999982826
> > Mean: 0.353602201999165
> > Median: 0.352992500003892
> > Std Dev: 0.00542072612850504
> > Starting test for consumer count 50
> >
> > Number of Consumers 50. Muliple Queues
> > Min: 0.315161000005901
> > Max: 0.425882000010461
> > Mean: 0.360078899599938
> > Median: 0.359610499988775
> > Std Dev: 0.00788422976924438
> > Starting test for consumer count 75
> >
> > Number of Consumers 75. Muliple Queues
> > Min: 0.342441000044346
> > Max: 0.66088400001172
> > Mean: 0.401721995466513
> > Median: 0.396242500049994
> > Std Dev: 0.0404559664668615
> > Starting test for consumer count 100
> >
> > Number of Consumers 100. Muliple Queues
> > Min: 0.352722999989055
> > Max: 3.99510599998757
> > Mean: 0.563622044800525
> > Median: 0.494796500017401
> > Std Dev: 0.413950797976057
> > Starting test for consumer count 300
> >
> > Number of Consumers 300. Muliple Queues
> > Min: 0.361888999934308
> > Max: 5.53048999991734
> > Mean: 1.91027370266765
> > Median: 1.8000390000525
> > Std Dev: 0.489824211293863
> > Starting test for consumer count 600
> >
> > Number of Consumers 600. Muliple Queues
> > Min: 0.335149999940768
> > Max: 10.6164910000516
> > Mean: 4.52802392866641
> > Median: 4.35808100004215
> > Std Dev: 0.840368954779232
> > Starting test for consumer count 900
> >
> > Number of Consumers 900. Muliple Queues
> > Min: 0.639438000041991
> > Max: 18.2733670000453
> > Mean: 8.00563488822206
> > Median: 7.6759294999647
> > Std Dev: 1.38340937172684
> > Starting test for consumer count 1200
> >
> > Number of Consumers 1200. Muliple Queues
> > Min: 0.474138000048697
> > Max: 31.5018520000158
> > Mean: 12.8169781057334
> > Median: 12.2411614999873
> > Std Dev: 2.45701978986895
> > Starting test for consumer count 1500
> >
> > Number of Consumers 1500. Muliple Queues
> > Min: 3.1234959999565
> > Max: 48.7995179999853
> > Mean: 18.8858608815866
> > Median: 17.5380175000173
> > Std Dev: 4.1516799330252
> > Starting test for consumer count 1800
> >
> > Number of Consumers 1800. Muliple Queues
> > Min: 4.99818900006358
> > Max: 73.2436839999864
> > Mean: 27.1358068585671
> > Median: 25.4123435000074
> > Std Dev: 6.30049000845097
> > Starting test for consumer count 2400
> >
> > Number of Consumers 2400. Muliple Queues
> > Min: 0.319424999994226
> > Max: 114.78910699999
> > Mean: 46.0846290592237
> > Median: 44.3440699999919
> > Std Dev: 10.2871979782358
> >
> > TEST Case only 1 queue
> >
> > Number of Consumers 1. Only One Queue
> > Min: 0.0413880000123754
> > Max: 0.0445370000088587
> > Mean: 0.0416983800008893
> > Median: 0.041657000023406
> > Std Dev: 0.00042437742781418
> > Starting test for consumer count 10
> >
> > Number of Consumers 10. Only One Queue
> > Min: 0.0409169999184087
> > Max: 0.0494429999962449
> > Mean: 0.0419903019983321
> > Median: 0.0417659999802709
> > Std Dev: 0.000839524388489985
> > Starting test for consumer count 50
> >
> > Number of Consumers 50. Only One Queue
> > Min: 0.00652100006118417
> > Max: 0.0843779999995604
> > Mean: 0.0431237947992515
> > Median: 0.0423434999538586
> > Std Dev: 0.00470470800328101
> > Starting test for consumer count 75
> >
> > Number of Consumers 75. Only One Queue
> > Min: 0.00334199995268136
> > Max: 0.120109000010416
> > Mean: 0.0456681223996294
> > Median: 0.0435704999836161
> > Std Dev: 0.00729394094656864
> > Starting test for consumer count 100
> >
> > Number of Consumers 100. Only One Queue
> > Min: 0.00263900007121265
> > Max: 0.206331999972463
> > Mean: 0.051723164400761
> > Median: 0.0513750000391155
> > Std Dev: 0.0225837245735077
> > Starting test for consumer count 300
> >
> > Number of Consumers 300. Only One Queue
> > Min: 0.00258900003973395
> > Max: 1.01170199993066
> > Mean: 0.138241231733017
> > Median: 0.136385999969207
> > Std Dev: 0.0863229692434055
> > Starting test for consumer count 600
> >
> > Number of Consumers 600. Only One Queue
> > Min: 0.00214999995660037
> > Max: 3.27785699989181
> > Mean: 0.274939405133063
> > Median: 0.256795499997679
> > Std Dev: 0.237695097382708
> > Starting test for consumer count 900
> >
> > Number of Consumers 900. Only One Queue
> > Min: 0.00206800003070384
> > Max: 31.7313950000098
> > Mean: 0.5553230254
> > Median: 0.338199999998324
> > Std Dev: 1.14882073602057
> > Starting test for consumer count 1200
> >
> > Number of Consumers 1200. Only One Queue
> > Min: 0.00257100001908839
> > Max: 49.8629720000317
> > Mean: 0.912980378683317
> > Median: 0.393762999970932
> > Std Dev: 2.87091387484458
> > Starting test for consumer count 1500
> >
> > Number of Consumers 1500. Only One Queue
> > Min: 0.00201100006233901
> > Max: 74.3607440000633
> > Mean: 1.19311908142647
> > Median: 0.205018000095152
> > Std Dev: 4.4037236439348
> > Starting test for consumer count 1800
> >
> > Number of Consumers 1800. Only One Queue
> > Min: 0.00196300004608929
> > Max: 84.4792379999999
> > Mean: 1.29789674880008
> > Median: 0.117239500046707
> > Std Dev: 5.19232074252423
> > Starting test for consumer count 2400
> >
> > Number of Consumers 2400. Only One Queue
> > Min: 0.00200599990785122
> > Max: 124.155756999971
> > Mean: 1.77886690554984
> > Median: 0.101840000017546
> > Std Dev: 8.38169615533614
> >
> >
>=20
>=20
>=20
> --
> http://blog.garytully.com
> http://fusesource.com