activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Carlson <jcarl...@e-dialog.com>
Subject RE: Scaleability problems with queue subscriptors
Date Thu, 05 May 2011 20:20:18 GMT
We are using the STOMP protocol which doesn't support that. I was curious if there might be
any settings server side which would help with the scalability of concurrent subscribes?

> -----Original Message-----
> From: Gary Tully [mailto:gary.tully@gmail.com]
> Sent: Wednesday, May 04, 2011 5:39 PM
> To: users@activemq.apache.org
> Subject: Re: Scaleability problems with queue subscriptors
> 
> have you tried to use prefetch=0 on the work queue, so the next
> message will not be dispatched till you issue another receive call
> rather than when the ack occurs.
> 
> On 4 May 2011 21:29, Josh Carlson <jcarlson@e-dialog.com> wrote:
> > Hi Gary,
> >
> > Thanks for the response. We've decided it would be easy for us to
> partition our consumers such that they each consumer operates on only
> one queue. However, the model we are using retrieves a message from one
> queue (the job queue), then grabs something to do from another queue
> (the work queue), once it retrieves the message from the work queue it
> acknowledges the job queue and goes and does its work. However, since
> another message is dispatched once the ack is done and the work can take
> a long time (potentially infinite) we unsubscribe.  Subsequently, once
> the work is done the consumer needs to go subscribe and retrieve another
> message.
> >
> > Switching to one queue helps when there is no or few messages.
> However, it is not scaling when there are plenty of messages due to the
> way we need to subscribe/unsubscribe. Do you have any suggestions on how
> we might be able to scale this given our current architecture?
> >
> > -Josh
> >
> >> -----Original Message-----
> >> From: Gary Tully [mailto:gary.tully@gmail.com]
> >> Sent: Friday, April 29, 2011 6:27 PM
> >> To: users@activemq.apache.org
> >> Subject: Re: Scaleability problems with queue subscriptors
> >>
> >> setting up a consumer is a little expensive, have a look at using a
> >> composite destination so that you can subscribe to all destinations
> at
> >> once.
> >> Also, there is a delay between new consumer registration and async
> >> dispatch, so waiting a few seconds before unsubscribe is necessary.
> >>
> >> http://activemq.apache.org/composite-destinations.html
> >>
> >> On 28 April 2011 23:41, Josh Carlson <jcarlson@e-dialog.com> wrote:
> >> > We are using a shared file system Master/Slave for the broker.
> Version
> >> 5.4.2. Our clients use the STOMP protocol. We use client
> >> acknowledgements and communicate synchronously with the broker (using
> >> receipts). We set prefetch to 1 in our subscriptions. Our clients
> >> iterate over several queues, subscribing, checking for messages
> (timeout
> >> of 50ms), and if one isn't available it un-subscribes and goes to the
> >> next queue. There are almost always cases where there are no messages
> in
> >> the queues. We ran into a problem where our application slowed down
> to a
> >> crawl when we deployed additional clients and I've narrowed it down
> to
> >> the fact that most of the time when we subscribed to the queue and
> then
> >> asked if a message was ready it wouldn't be even though there were
> >> messages in the queue. My assumption is that it is taking some time
> to
> >> dispatch the message.
> >> >
> >> > Is there some configuration parameters I might want to set to help
> >> with this problem? Or is this type of use just not going to scale?
> >> >
> >> > Here is some benchmark data. Each test, creates N consumers, but
> >> before they are allowed to start it enqueues 50*N messages for the
> >> consumer into *one* queue. The first set of metrics is for the case
> >> where the consumers are iterating over 6 different queues (even
> though
> >> there is only data in one). The second set of metrics we ONLY have 1
> >> queue ... in this case the client only subscribes and un-subscribes
> once
> >> except in the case where a message 'isn't ready' in that 50ms (in
> which
> >> case it re-subscribes to the same queue). The metrics capture the
> entire
> >> process. getNextMessage, iterates over the queues, doing the
> >> subscribes/un-subscribes, receipts etc ...
> >> >
> >> > Note that in the 6 queue case time degrades once you have 100
> >> consumers. In the other case it degrades after 100 but we never see a
> >> Median greater than 206ms.
> >> >
> >> > TEST Case 6 Queues ... 5 of which are empty (note that in this
> first
> >> case since 5 queues are empty one expects at least 250ms to poll
> those 5
> >> empty queues). Times are in seconds.
> >> >
> >> > Number of Consumers 1. Muliple Queues
> >> > Min: 0.349334999918938
> >> > Max: 0.368788999971002
> >> > Mean: 0.350222800001502
> >> > Median: 0.349644500005525
> >> > Std Dev: 0.00271797410606451
> >> > Starting test for consumer count 10
> >> >
> >> > Number of Consumers 10. Muliple Queues
> >> > Min: 0.349282000097446
> >> > Max: 0.394184999982826
> >> > Mean: 0.353602201999165
> >> > Median: 0.352992500003892
> >> > Std Dev: 0.00542072612850504
> >> > Starting test for consumer count 50
> >> >
> >> > Number of Consumers 50. Muliple Queues
> >> > Min: 0.315161000005901
> >> > Max: 0.425882000010461
> >> > Mean: 0.360078899599938
> >> > Median: 0.359610499988775
> >> > Std Dev: 0.00788422976924438
> >> > Starting test for consumer count 75
> >> >
> >> > Number of Consumers 75. Muliple Queues
> >> > Min: 0.342441000044346
> >> > Max: 0.66088400001172
> >> > Mean: 0.401721995466513
> >> > Median: 0.396242500049994
> >> > Std Dev: 0.0404559664668615
> >> > Starting test for consumer count 100
> >> >
> >> > Number of Consumers 100. Muliple Queues
> >> > Min: 0.352722999989055
> >> > Max: 3.99510599998757
> >> > Mean: 0.563622044800525
> >> > Median: 0.494796500017401
> >> > Std Dev: 0.413950797976057
> >> > Starting test for consumer count 300
> >> >
> >> > Number of Consumers 300. Muliple Queues
> >> > Min: 0.361888999934308
> >> > Max: 5.53048999991734
> >> > Mean: 1.91027370266765
> >> > Median: 1.8000390000525
> >> > Std Dev: 0.489824211293863
> >> > Starting test for consumer count 600
> >> >
> >> > Number of Consumers 600. Muliple Queues
> >> > Min: 0.335149999940768
> >> > Max: 10.6164910000516
> >> > Mean: 4.52802392866641
> >> > Median: 4.35808100004215
> >> > Std Dev: 0.840368954779232
> >> > Starting test for consumer count 900
> >> >
> >> > Number of Consumers 900. Muliple Queues
> >> > Min: 0.639438000041991
> >> > Max: 18.2733670000453
> >> > Mean: 8.00563488822206
> >> > Median: 7.6759294999647
> >> > Std Dev: 1.38340937172684
> >> > Starting test for consumer count 1200
> >> >
> >> > Number of Consumers 1200. Muliple Queues
> >> > Min: 0.474138000048697
> >> > Max: 31.5018520000158
> >> > Mean: 12.8169781057334
> >> > Median: 12.2411614999873
> >> > Std Dev: 2.45701978986895
> >> > Starting test for consumer count 1500
> >> >
> >> > Number of Consumers 1500. Muliple Queues
> >> > Min: 3.1234959999565
> >> > Max: 48.7995179999853
> >> > Mean: 18.8858608815866
> >> > Median: 17.5380175000173
> >> > Std Dev: 4.1516799330252
> >> > Starting test for consumer count 1800
> >> >
> >> > Number of Consumers 1800. Muliple Queues
> >> > Min: 4.99818900006358
> >> > Max: 73.2436839999864
> >> > Mean: 27.1358068585671
> >> > Median: 25.4123435000074
> >> > Std Dev: 6.30049000845097
> >> > Starting test for consumer count 2400
> >> >
> >> > Number of Consumers 2400. Muliple Queues
> >> > Min: 0.319424999994226
> >> > Max: 114.78910699999
> >> > Mean: 46.0846290592237
> >> > Median: 44.3440699999919
> >> > Std Dev: 10.2871979782358
> >> >
> >> > TEST Case only 1 queue
> >> >
> >> > Number of Consumers 1. Only One Queue
> >> > Min: 0.0413880000123754
> >> > Max: 0.0445370000088587
> >> > Mean: 0.0416983800008893
> >> > Median: 0.041657000023406
> >> > Std Dev: 0.00042437742781418
> >> > Starting test for consumer count 10
> >> >
> >> > Number of Consumers 10. Only One Queue
> >> > Min: 0.0409169999184087
> >> > Max: 0.0494429999962449
> >> > Mean: 0.0419903019983321
> >> > Median: 0.0417659999802709
> >> > Std Dev: 0.000839524388489985
> >> > Starting test for consumer count 50
> >> >
> >> > Number of Consumers 50. Only One Queue
> >> > Min: 0.00652100006118417
> >> > Max: 0.0843779999995604
> >> > Mean: 0.0431237947992515
> >> > Median: 0.0423434999538586
> >> > Std Dev: 0.00470470800328101
> >> > Starting test for consumer count 75
> >> >
> >> > Number of Consumers 75. Only One Queue
> >> > Min: 0.00334199995268136
> >> > Max: 0.120109000010416
> >> > Mean: 0.0456681223996294
> >> > Median: 0.0435704999836161
> >> > Std Dev: 0.00729394094656864
> >> > Starting test for consumer count 100
> >> >
> >> > Number of Consumers 100. Only One Queue
> >> > Min: 0.00263900007121265
> >> > Max: 0.206331999972463
> >> > Mean: 0.051723164400761
> >> > Median: 0.0513750000391155
> >> > Std Dev: 0.0225837245735077
> >> > Starting test for consumer count 300
> >> >
> >> > Number of Consumers 300. Only One Queue
> >> > Min: 0.00258900003973395
> >> > Max: 1.01170199993066
> >> > Mean: 0.138241231733017
> >> > Median: 0.136385999969207
> >> > Std Dev: 0.0863229692434055
> >> > Starting test for consumer count 600
> >> >
> >> > Number of Consumers 600. Only One Queue
> >> > Min: 0.00214999995660037
> >> > Max: 3.27785699989181
> >> > Mean: 0.274939405133063
> >> > Median: 0.256795499997679
> >> > Std Dev: 0.237695097382708
> >> > Starting test for consumer count 900
> >> >
> >> > Number of Consumers 900. Only One Queue
> >> > Min: 0.00206800003070384
> >> > Max: 31.7313950000098
> >> > Mean: 0.5553230254
> >> > Median: 0.338199999998324
> >> > Std Dev: 1.14882073602057
> >> > Starting test for consumer count 1200
> >> >
> >> > Number of Consumers 1200. Only One Queue
> >> > Min: 0.00257100001908839
> >> > Max: 49.8629720000317
> >> > Mean: 0.912980378683317
> >> > Median: 0.393762999970932
> >> > Std Dev: 2.87091387484458
> >> > Starting test for consumer count 1500
> >> >
> >> > Number of Consumers 1500. Only One Queue
> >> > Min: 0.00201100006233901
> >> > Max: 74.3607440000633
> >> > Mean: 1.19311908142647
> >> > Median: 0.205018000095152
> >> > Std Dev: 4.4037236439348
> >> > Starting test for consumer count 1800
> >> >
> >> > Number of Consumers 1800. Only One Queue
> >> > Min: 0.00196300004608929
> >> > Max: 84.4792379999999
> >> > Mean: 1.29789674880008
> >> > Median: 0.117239500046707
> >> > Std Dev: 5.19232074252423
> >> > Starting test for consumer count 2400
> >> >
> >> > Number of Consumers 2400. Only One Queue
> >> > Min: 0.00200599990785122
> >> > Max: 124.155756999971
> >> > Mean: 1.77886690554984
> >> > Median: 0.101840000017546
> >> > Std Dev: 8.38169615533614
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> http://blog.garytully.com
> >> http://fusesource.com
> >
> 
> 
> 
> --
> http://blog.garytully.com
> http://fusesource.com

Mime
View raw message