qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gordon Sim <g...@redhat.com>
Subject Re: Load balancing messages to consumers that have different performances characteristics
Date Mon, 07 Mar 2011 16:28:50 GMT
On 03/07/2011 02:25 PM, Julien Lavigne du Cadet wrote:
> Hi,
>
> I have the following scenario :
>
> I'm pushing 100 messages to a queue shared by 2 consumers. Both
> subscribers subscribe to the queue in pre-acquire mode and explicit
> mode. After each message is handled, each subscriber accepts the message
> to remove it from the queue. The pseudo codes look likes that :
>
> OnMessageTransfer(message) :
>
>      DoSomethingWithMessage(message)
>
>      Session.MessageAccept(message)
>
> The messages are load balanced correctly, each message is processed once
> and only once, but we discovered that it doesn't take into account the
> processing time for each consumer. For instance, let's assume that
> consumer A is taking 50ms to process a message and consumer B is taking
> 5seconds. Ideally, consumer B should start processing 1 message and in
> the meantime, consumer A should process the 99 others. However, what
> happens is that consumer B will actually process 25 messages in 50
> seconds while consumer A will process the 75 others in ~4 seconds and
> will idle. The client api seems to prefetch the messages, which is
> clearly non optimal in this situation.
>
> How can we solve this problem?

I'm not terribly familiar with the API you are using. However what you 
want to do at the protocol level is alter the credit for the 
subscriptions. A quick scan through the code shows the client offering a 
MessageSubscribe convenience wrapper:

    MessageSubscribe(queue, queue, MessageAcceptMode.EXPLICIT, 
MessageAcquireMode.PRE_ACQUIRED, null, 0, null);
    // issue credits
    MessageSetFlowMode(queue, MessageFlowMode.WINDOW);
    MessageFlow(queue, MessageCreditUnit.BYTE, MESSAGE_FLOW_MAX_BYTES);
    MessageFlow(queue, MessageCreditUnit.MESSAGE, 10000); 


That last line is issuing credit for 10000 messages for each 
subscription. If instead you allocated 1 message then the prefetch is 
reduced to one message.

The credit window is moved not on MessageAccept but on SessionCompleted. 
I can't tell from a quick glance how the API exposes control over 
completion.

Failing some solution for signalling completion, you could always change 
the flow mode to MessageFlowMode.CREDIT (instead of 
MessageFlowMode.WINDOW) and explicitly allocate more credit with 
MessageFlow(queue, MessageCreditUnit.MESSAGE, 1).

Hope this helps a little, sorry I can't be more precise about that 
particular client!

> We're using Qpid cpp 0.5 and the fully managed c# 0-10 client API, not
> the cpp bindings (but my understanding is that this behavior is not
> linked to the implementation of the API)
>
> Message cross posted on stack overflow if someone prefers to answer
> there :
>
> http://stackoverflow.com/questions/5220808/qpid-load-balancing-messages-
> to-consumers-that-have-different-performances-char
> <http://stackoverflow.com/questions/5220808/qpid-load-balancing-messages
> -to-consumers-that-have-different-performances-char>
>
> Regards,
>
> Julien
>
>
>
>
> -------------------------------------------------------------------------------------------
> ABC arbitrage, partenaire officiel du skipper Jean-Pierre Dick (bateau Virbac Paprec
3), engagé avec Loïc Peyron sur la Barcelona World Race 2011, course autour du monde en
double sans escale // ABC arbitrage, official partner of skipper Jean-Pierre Dick (Virbac
Paprec 3 boat), racing with Loïc Peyron in the 2011 edition of the Barcelona World Race (double-handed
non-stop round-the-world race).
> P  Please consider your environmental responsibility before printing this email
> *********************************************************************************
> Ce message peut contenir des informations confidentielles.
> Les idees et opinions presentees dans ce message
> sont celles de son auteur, et ne representent pas necessairement
> celles du groupe ABC arbitrage.
> Au cas ou il ne vous serait pas destine,
> merci d'en aviser l'expediteur immediatement et de le supprimer.
>
> This message may contain confidential information.
> Any views or opinions presented are solely those of its author
> and do not necessarily represent those of ABC arbitrage.
> If you are not the intended recipient,
> please notify the sender immediately and delete it.
> *********************************************************************************
>
>


---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org


Mime
View raw message