activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Sitsky <s...@nuix.com>
Subject Queue performance from recent changes
Date Wed, 20 Feb 2008 06:31:39 GMT
Hi Rob,

I like the new changes, but with the changes as they are, for my 
application for one of my benchmarks, it takes twice as long to complete.

I believe the culprit for this is that when the new code can't find a 
consumer which is not full, the broker chooses the consumer with the 
lowest dispatch queue size.

In my application, since I have a prefetch size of 1, and have 
longish-running transactions, the dispatch queue size is not indicative 
of the current load for that consumer.  As a result, I think this is 
what is responsible for poor load-balancing in my case.

For applications which commit() after each processed message, I am sure 
this wouldn't be the case.  In some ways, reverting to the old behaviour 
of adding the pending message to all consumers might lead to better load 
balancing with this code.

However - I think it is better if the consumers can decide when they 
want more messages rather than the broker pushing messages at them? 
I've attached a patch which demonstrates this.  When LAZY_DISPATCH is 
set to true (set via a system property for now for testing purposes) 
this changes the behaviour slightly.

The basic idea is pageInMessages() only pages in the minimum number of 
messages that can be dispatched immediately to non-full consumers. 
Whenever a consumer acks a message, which updates its prefetch size, we 
make sure Queue.wakeup() is called so that the consumer will receive new 
messages.

With this change in effect - I see slightly faster or almost the same 
times with the previous benchmark.  However memory usage on the broker 
is far better, as the pending queues for each consumer is either 0 or 
very small.

What do you think?  I guess there are better ways of doing this.

I am doing a large overnight run with 16 consumers, so we'll see how the 
  performance goes.

You'll also notice in the patch, that in Queue.addSubscriber(), I 
thought there didn't seem to be any need for adding a message to a new 
consumer if the message has already been locked by another consumer?

Cheers,
David

Rob Davies wrote:
> Hi David,
> 
> please let us know if these changes helps/hinders your app!
> 
> cheers,
> 
> Rob
> On 19 Feb 2008, at 08:32, David Sitsky wrote:
> 
>>>> If what I said above is true, then the immediately above if 
>>>> statement needs to be moved outside its enclosing if - otherwise it 
>>>> only gets executed when targets != null.  We'd want this to execute 
>>>> if we found a matching target wouldn't we?
>>> Don't think so? We only want the message going to  one subscription? 
>>> I may have misunderstood what you mean!
>> Yes - ignore what I said, I had my wires crossed.
>>
>> Cheers,
>> David
>>


-- 
Cheers,
David

Nuix Pty Ltd
Suite 79, 89 Jones St, Ultimo NSW 2007, Australia    Ph: +61 2 9280 0699
Web: http://www.nuix.com                            Fax: +61 2 9212 6902

Mime
View raw message