activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Carlson <jcarl...@e-dialog.com>
Subject Scaleability problems with queue subscriptors
Date Thu, 28 Apr 2011 22:41:57 GMT
We are using a shared file system Master/Slave for the broker. Version 5.4.2. Our clients use
the STOMP protocol. We use client acknowledgements and communicate synchronously with the
broker (using receipts). We set prefetch to 1 in our subscriptions. Our clients iterate over
several queues, subscribing, checking for messages (timeout of 50ms), and if one isn't available
it un-subscribes and goes to the next queue. There are almost always cases where there are
no messages in the queues. We ran into a problem where our application slowed down to a crawl
when we deployed additional clients and I've narrowed it down to the fact that most of the
time when we subscribed to the queue and then asked if a message was ready it wouldn't be
even though there were messages in the queue. My assumption is that it is taking some time
to dispatch the message.

Is there some configuration parameters I might want to set to help with this problem? Or is
this type of use just not going to scale?

Here is some benchmark data. Each test, creates N consumers, but before they are allowed to
start it enqueues 50*N messages for the consumer into *one* queue. The first set of metrics
is for the case where the consumers are iterating over 6 different queues (even though there
is only data in one). The second set of metrics we ONLY have 1 queue ... in this case the
client only subscribes and un-subscribes once except in the case where a message 'isn't ready'
in that 50ms (in which case it re-subscribes to the same queue). The metrics capture the entire
process. getNextMessage, iterates over the queues, doing the subscribes/un-subscribes, receipts
etc ...

Note that in the 6 queue case time degrades once you have 100 consumers. In the other case
it degrades after 100 but we never see a Median greater than 206ms.

TEST Case 6 Queues ... 5 of which are empty (note that in this first case since 5 queues are
empty one expects at least 250ms to poll those 5 empty queues). Times are in seconds.

Number of Consumers 1. Muliple Queues
Min: 0.349334999918938
Max: 0.368788999971002
Mean: 0.350222800001502
Median: 0.349644500005525 
Std Dev: 0.00271797410606451
Starting test for consumer count 10

Number of Consumers 10. Muliple Queues
Min: 0.349282000097446
Max: 0.394184999982826
Mean: 0.353602201999165
Median: 0.352992500003892 
Std Dev: 0.00542072612850504
Starting test for consumer count 50

Number of Consumers 50. Muliple Queues
Min: 0.315161000005901
Max: 0.425882000010461
Mean: 0.360078899599938
Median: 0.359610499988775 
Std Dev: 0.00788422976924438
Starting test for consumer count 75

Number of Consumers 75. Muliple Queues
Min: 0.342441000044346
Max: 0.66088400001172
Mean: 0.401721995466513
Median: 0.396242500049994 
Std Dev: 0.0404559664668615
Starting test for consumer count 100

Number of Consumers 100. Muliple Queues
Min: 0.352722999989055
Max: 3.99510599998757
Mean: 0.563622044800525
Median: 0.494796500017401 
Std Dev: 0.413950797976057
Starting test for consumer count 300

Number of Consumers 300. Muliple Queues
Min: 0.361888999934308
Max: 5.53048999991734
Mean: 1.91027370266765
Median: 1.8000390000525 
Std Dev: 0.489824211293863
Starting test for consumer count 600

Number of Consumers 600. Muliple Queues
Min: 0.335149999940768
Max: 10.6164910000516
Mean: 4.52802392866641
Median: 4.35808100004215 
Std Dev: 0.840368954779232
Starting test for consumer count 900

Number of Consumers 900. Muliple Queues
Min: 0.639438000041991
Max: 18.2733670000453
Mean: 8.00563488822206
Median: 7.6759294999647 
Std Dev: 1.38340937172684
Starting test for consumer count 1200

Number of Consumers 1200. Muliple Queues
Min: 0.474138000048697
Max: 31.5018520000158
Mean: 12.8169781057334
Median: 12.2411614999873 
Std Dev: 2.45701978986895
Starting test for consumer count 1500

Number of Consumers 1500. Muliple Queues
Min: 3.1234959999565
Max: 48.7995179999853
Mean: 18.8858608815866
Median: 17.5380175000173 
Std Dev: 4.1516799330252
Starting test for consumer count 1800

Number of Consumers 1800. Muliple Queues
Min: 4.99818900006358
Max: 73.2436839999864
Mean: 27.1358068585671
Median: 25.4123435000074 
Std Dev: 6.30049000845097
Starting test for consumer count 2400

Number of Consumers 2400. Muliple Queues
Min: 0.319424999994226
Max: 114.78910699999
Mean: 46.0846290592237
Median: 44.3440699999919 
Std Dev: 10.2871979782358

TEST Case only 1 queue

Number of Consumers 1. Only One Queue
Min: 0.0413880000123754
Max: 0.0445370000088587
Mean: 0.0416983800008893
Median: 0.041657000023406 
Std Dev: 0.00042437742781418
Starting test for consumer count 10

Number of Consumers 10. Only One Queue
Min: 0.0409169999184087
Max: 0.0494429999962449
Mean: 0.0419903019983321
Median: 0.0417659999802709 
Std Dev: 0.000839524388489985
Starting test for consumer count 50

Number of Consumers 50. Only One Queue
Min: 0.00652100006118417
Max: 0.0843779999995604
Mean: 0.0431237947992515
Median: 0.0423434999538586 
Std Dev: 0.00470470800328101
Starting test for consumer count 75

Number of Consumers 75. Only One Queue
Min: 0.00334199995268136
Max: 0.120109000010416
Mean: 0.0456681223996294
Median: 0.0435704999836161 
Std Dev: 0.00729394094656864
Starting test for consumer count 100

Number of Consumers 100. Only One Queue
Min: 0.00263900007121265
Max: 0.206331999972463
Mean: 0.051723164400761
Median: 0.0513750000391155 
Std Dev: 0.0225837245735077
Starting test for consumer count 300

Number of Consumers 300. Only One Queue
Min: 0.00258900003973395
Max: 1.01170199993066
Mean: 0.138241231733017
Median: 0.136385999969207 
Std Dev: 0.0863229692434055
Starting test for consumer count 600

Number of Consumers 600. Only One Queue
Min: 0.00214999995660037
Max: 3.27785699989181
Mean: 0.274939405133063
Median: 0.256795499997679 
Std Dev: 0.237695097382708
Starting test for consumer count 900

Number of Consumers 900. Only One Queue
Min: 0.00206800003070384
Max: 31.7313950000098
Mean: 0.5553230254
Median: 0.338199999998324 
Std Dev: 1.14882073602057
Starting test for consumer count 1200

Number of Consumers 1200. Only One Queue
Min: 0.00257100001908839
Max: 49.8629720000317
Mean: 0.912980378683317
Median: 0.393762999970932 
Std Dev: 2.87091387484458
Starting test for consumer count 1500

Number of Consumers 1500. Only One Queue
Min: 0.00201100006233901
Max: 74.3607440000633
Mean: 1.19311908142647
Median: 0.205018000095152 
Std Dev: 4.4037236439348
Starting test for consumer count 1800

Number of Consumers 1800. Only One Queue
Min: 0.00196300004608929
Max: 84.4792379999999
Mean: 1.29789674880008
Median: 0.117239500046707 
Std Dev: 5.19232074252423
Starting test for consumer count 2400

Number of Consumers 2400. Only One Queue
Min: 0.00200599990785122
Max: 124.155756999971
Mean: 1.77886690554984
Median: 0.101840000017546 
Std Dev: 8.38169615533614


Mime
View raw message