activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <gary.tu...@gmail.com>
Subject Re: Scaleability problems with queue subscriptors
Date Fri, 29 Apr 2011 22:27:25 GMT
setting up a consumer is a little expensive, have a look at using a
composite destination so that you can subscribe to all destinations at
once.
Also, there is a delay between new consumer registration and async
dispatch, so waiting a few seconds before unsubscribe is necessary.

http://activemq.apache.org/composite-destinations.html

On 28 April 2011 23:41, Josh Carlson <jcarlson@e-dialog.com> wrote:
> We are using a shared file system Master/Slave for the broker. Version 5.4.2. Our clients
use the STOMP protocol. We use client acknowledgements and communicate synchronously with
the broker (using receipts). We set prefetch to 1 in our subscriptions. Our clients iterate
over several queues, subscribing, checking for messages (timeout of 50ms), and if one isn't
available it un-subscribes and goes to the next queue. There are almost always cases where
there are no messages in the queues. We ran into a problem where our application slowed down
to a crawl when we deployed additional clients and I've narrowed it down to the fact that
most of the time when we subscribed to the queue and then asked if a message was ready it
wouldn't be even though there were messages in the queue. My assumption is that it is taking
some time to dispatch the message.
>
> Is there some configuration parameters I might want to set to help with this problem?
Or is this type of use just not going to scale?
>
> Here is some benchmark data. Each test, creates N consumers, but before they are allowed
to start it enqueues 50*N messages for the consumer into *one* queue. The first set of metrics
is for the case where the consumers are iterating over 6 different queues (even though there
is only data in one). The second set of metrics we ONLY have 1 queue ... in this case the
client only subscribes and un-subscribes once except in the case where a message 'isn't ready'
in that 50ms (in which case it re-subscribes to the same queue). The metrics capture the entire
process. getNextMessage, iterates over the queues, doing the subscribes/un-subscribes, receipts
etc ...
>
> Note that in the 6 queue case time degrades once you have 100 consumers. In the other
case it degrades after 100 but we never see a Median greater than 206ms.
>
> TEST Case 6 Queues ... 5 of which are empty (note that in this first case since 5 queues
are empty one expects at least 250ms to poll those 5 empty queues). Times are in seconds.
>
> Number of Consumers 1. Muliple Queues
> Min: 0.349334999918938
> Max: 0.368788999971002
> Mean: 0.350222800001502
> Median: 0.349644500005525
> Std Dev: 0.00271797410606451
> Starting test for consumer count 10
>
> Number of Consumers 10. Muliple Queues
> Min: 0.349282000097446
> Max: 0.394184999982826
> Mean: 0.353602201999165
> Median: 0.352992500003892
> Std Dev: 0.00542072612850504
> Starting test for consumer count 50
>
> Number of Consumers 50. Muliple Queues
> Min: 0.315161000005901
> Max: 0.425882000010461
> Mean: 0.360078899599938
> Median: 0.359610499988775
> Std Dev: 0.00788422976924438
> Starting test for consumer count 75
>
> Number of Consumers 75. Muliple Queues
> Min: 0.342441000044346
> Max: 0.66088400001172
> Mean: 0.401721995466513
> Median: 0.396242500049994
> Std Dev: 0.0404559664668615
> Starting test for consumer count 100
>
> Number of Consumers 100. Muliple Queues
> Min: 0.352722999989055
> Max: 3.99510599998757
> Mean: 0.563622044800525
> Median: 0.494796500017401
> Std Dev: 0.413950797976057
> Starting test for consumer count 300
>
> Number of Consumers 300. Muliple Queues
> Min: 0.361888999934308
> Max: 5.53048999991734
> Mean: 1.91027370266765
> Median: 1.8000390000525
> Std Dev: 0.489824211293863
> Starting test for consumer count 600
>
> Number of Consumers 600. Muliple Queues
> Min: 0.335149999940768
> Max: 10.6164910000516
> Mean: 4.52802392866641
> Median: 4.35808100004215
> Std Dev: 0.840368954779232
> Starting test for consumer count 900
>
> Number of Consumers 900. Muliple Queues
> Min: 0.639438000041991
> Max: 18.2733670000453
> Mean: 8.00563488822206
> Median: 7.6759294999647
> Std Dev: 1.38340937172684
> Starting test for consumer count 1200
>
> Number of Consumers 1200. Muliple Queues
> Min: 0.474138000048697
> Max: 31.5018520000158
> Mean: 12.8169781057334
> Median: 12.2411614999873
> Std Dev: 2.45701978986895
> Starting test for consumer count 1500
>
> Number of Consumers 1500. Muliple Queues
> Min: 3.1234959999565
> Max: 48.7995179999853
> Mean: 18.8858608815866
> Median: 17.5380175000173
> Std Dev: 4.1516799330252
> Starting test for consumer count 1800
>
> Number of Consumers 1800. Muliple Queues
> Min: 4.99818900006358
> Max: 73.2436839999864
> Mean: 27.1358068585671
> Median: 25.4123435000074
> Std Dev: 6.30049000845097
> Starting test for consumer count 2400
>
> Number of Consumers 2400. Muliple Queues
> Min: 0.319424999994226
> Max: 114.78910699999
> Mean: 46.0846290592237
> Median: 44.3440699999919
> Std Dev: 10.2871979782358
>
> TEST Case only 1 queue
>
> Number of Consumers 1. Only One Queue
> Min: 0.0413880000123754
> Max: 0.0445370000088587
> Mean: 0.0416983800008893
> Median: 0.041657000023406
> Std Dev: 0.00042437742781418
> Starting test for consumer count 10
>
> Number of Consumers 10. Only One Queue
> Min: 0.0409169999184087
> Max: 0.0494429999962449
> Mean: 0.0419903019983321
> Median: 0.0417659999802709
> Std Dev: 0.000839524388489985
> Starting test for consumer count 50
>
> Number of Consumers 50. Only One Queue
> Min: 0.00652100006118417
> Max: 0.0843779999995604
> Mean: 0.0431237947992515
> Median: 0.0423434999538586
> Std Dev: 0.00470470800328101
> Starting test for consumer count 75
>
> Number of Consumers 75. Only One Queue
> Min: 0.00334199995268136
> Max: 0.120109000010416
> Mean: 0.0456681223996294
> Median: 0.0435704999836161
> Std Dev: 0.00729394094656864
> Starting test for consumer count 100
>
> Number of Consumers 100. Only One Queue
> Min: 0.00263900007121265
> Max: 0.206331999972463
> Mean: 0.051723164400761
> Median: 0.0513750000391155
> Std Dev: 0.0225837245735077
> Starting test for consumer count 300
>
> Number of Consumers 300. Only One Queue
> Min: 0.00258900003973395
> Max: 1.01170199993066
> Mean: 0.138241231733017
> Median: 0.136385999969207
> Std Dev: 0.0863229692434055
> Starting test for consumer count 600
>
> Number of Consumers 600. Only One Queue
> Min: 0.00214999995660037
> Max: 3.27785699989181
> Mean: 0.274939405133063
> Median: 0.256795499997679
> Std Dev: 0.237695097382708
> Starting test for consumer count 900
>
> Number of Consumers 900. Only One Queue
> Min: 0.00206800003070384
> Max: 31.7313950000098
> Mean: 0.5553230254
> Median: 0.338199999998324
> Std Dev: 1.14882073602057
> Starting test for consumer count 1200
>
> Number of Consumers 1200. Only One Queue
> Min: 0.00257100001908839
> Max: 49.8629720000317
> Mean: 0.912980378683317
> Median: 0.393762999970932
> Std Dev: 2.87091387484458
> Starting test for consumer count 1500
>
> Number of Consumers 1500. Only One Queue
> Min: 0.00201100006233901
> Max: 74.3607440000633
> Mean: 1.19311908142647
> Median: 0.205018000095152
> Std Dev: 4.4037236439348
> Starting test for consumer count 1800
>
> Number of Consumers 1800. Only One Queue
> Min: 0.00196300004608929
> Max: 84.4792379999999
> Mean: 1.29789674880008
> Median: 0.117239500046707
> Std Dev: 5.19232074252423
> Starting test for consumer count 2400
>
> Number of Consumers 2400. Only One Queue
> Min: 0.00200599990785122
> Max: 124.155756999971
> Mean: 1.77886690554984
> Median: 0.101840000017546
> Std Dev: 8.38169615533614
>
>



-- 
http://blog.garytully.com
http://fusesource.com

Mime
View raw message