From users-return-3861-apmail-qpid-users-archive=qpid.apache.org@qpid.apache.org Tue Mar 08 08:39:41 2011 Return-Path: Delivered-To: apmail-qpid-users-archive@www.apache.org Received: (qmail 13932 invoked from network); 8 Mar 2011 08:39:41 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 8 Mar 2011 08:39:41 -0000 Received: (qmail 74089 invoked by uid 500); 8 Mar 2011 08:39:41 -0000 Delivered-To: apmail-qpid-users-archive@qpid.apache.org Received: (qmail 74031 invoked by uid 500); 8 Mar 2011 08:39:40 -0000 Mailing-List: contact users-help@qpid.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@qpid.apache.org Delivered-To: mailing list users@qpid.apache.org Received: (qmail 74023 invoked by uid 99); 8 Mar 2011 08:39:40 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 08:39:40 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [213.41.18.43] (HELO vm-abcsmtp.abc-arbitrage.com) (213.41.18.43) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 08:39:33 +0000 Received: from vm-abcexch2003.abc-arbitrage.fr (unverified [192.168.191.14]) by vm-abcsmtp.abc-arbitrage.com (Clearswift SMTPRS 5.4.0) with ESMTP id for ; Tue, 8 Mar 2011 09:39:12 +0100 x-mimeole: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Subject: RE: Load balancing messages to consumers that have different performances characteristics Date: Tue, 8 Mar 2011 09:39:12 +0100 Message-ID: <3BC56299C1B27B4FA2FF39164EE020870103F031@vm-abcexch2003.abc-arbitrage.fr> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Load balancing messages to consumers that have different performances characteristics Thread-Index: Acvc5Q2Ac4BDoreYQjit8EFK5K1xvQAhzQGA References: <3BC56299C1B27B4FA2FF39164EE020870103F015@vm-abcexch2003.abc-arbitrage.fr> <4D7507C2.9060301@redhat.com> From: "Julien Lavigne du Cadet" To: X-Virus-Checked: Checked by ClamAV on apache.org Thank you Gordon, it works perfectly. -----Message d'origine----- De=A0: Gordon Sim [mailto:gsim@redhat.com]=20 Envoy=E9=A0: lundi 7 mars 2011 17:29 =C0=A0: users@qpid.apache.org Objet=A0: Re: Load balancing messages to consumers that have different = performances characteristics On 03/07/2011 02:25 PM, Julien Lavigne du Cadet wrote: > Hi, > > I have the following scenario : > > I'm pushing 100 messages to a queue shared by 2 consumers. Both > subscribers subscribe to the queue in pre-acquire mode and explicit > mode. After each message is handled, each subscriber accepts the = message > to remove it from the queue. The pseudo codes look likes that : > > OnMessageTransfer(message) : > > DoSomethingWithMessage(message) > > Session.MessageAccept(message) > > The messages are load balanced correctly, each message is processed = once > and only once, but we discovered that it doesn't take into account the > processing time for each consumer. For instance, let's assume that > consumer A is taking 50ms to process a message and consumer B is = taking > 5seconds. Ideally, consumer B should start processing 1 message and in > the meantime, consumer A should process the 99 others. However, what > happens is that consumer B will actually process 25 messages in 50 > seconds while consumer A will process the 75 others in ~4 seconds and > will idle. The client api seems to prefetch the messages, which is > clearly non optimal in this situation. > > How can we solve this problem? I'm not terribly familiar with the API you are using. However what you=20 want to do at the protocol level is alter the credit for the=20 subscriptions. A quick scan through the code shows the client offering a = MessageSubscribe convenience wrapper: MessageSubscribe(queue, queue, MessageAcceptMode.EXPLICIT,=20 MessageAcquireMode.PRE_ACQUIRED, null, 0, null); // issue credits MessageSetFlowMode(queue, MessageFlowMode.WINDOW); MessageFlow(queue, MessageCreditUnit.BYTE, MESSAGE_FLOW_MAX_BYTES); MessageFlow(queue, MessageCreditUnit.MESSAGE, 10000);=20 That last line is issuing credit for 10000 messages for each=20 subscription. If instead you allocated 1 message then the prefetch is=20 reduced to one message. The credit window is moved not on MessageAccept but on SessionCompleted. = I can't tell from a quick glance how the API exposes control over=20 completion. Failing some solution for signalling completion, you could always change = the flow mode to MessageFlowMode.CREDIT (instead of=20 MessageFlowMode.WINDOW) and explicitly allocate more credit with=20 MessageFlow(queue, MessageCreditUnit.MESSAGE, 1). Hope this helps a little, sorry I can't be more precise about that=20 particular client! > We're using Qpid cpp 0.5 and the fully managed c# 0-10 client API, not > the cpp bindings (but my understanding is that this behavior is not > linked to the implementation of the API) > > Message cross posted on stack overflow if someone prefers to answer > there : > > = http://stackoverflow.com/questions/5220808/qpid-load-balancing-messages- > to-consumers-that-have-different-performances-char > = -to-consumers-that-have-different-performances-char> > > Regards, > > Julien > > > > > = -------------------------------------------------------------------------= ------------------ > ABC arbitrage, partenaire officiel du skipper Jean-Pierre Dick (bateau = Virbac Paprec 3), engag=E9 avec Lo=EFc Peyron sur la Barcelona World = Race 2011, course autour du monde en double sans escale // ABC = arbitrage, official partner of skipper Jean-Pierre Dick (Virbac Paprec 3 = boat), racing with Lo=EFc Peyron in the 2011 edition of the Barcelona = World Race (double-handed non-stop round-the-world race). > P Please consider your environmental responsibility before printing = this email > = *************************************************************************= ******** > Ce message peut contenir des informations confidentielles. > Les idees et opinions presentees dans ce message > sont celles de son auteur, et ne representent pas necessairement > celles du groupe ABC arbitrage. > Au cas ou il ne vous serait pas destine, > merci d'en aviser l'expediteur immediatement et de le supprimer. > > This message may contain confidential information. > Any views or opinions presented are solely those of its author > and do not necessarily represent those of ABC arbitrage. > If you are not the intended recipient, > please notify the sender immediately and delete it. > = *************************************************************************= ******** > > --------------------------------------------------------------------- Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:users-subscribe@qpid.apache.org --------------------------------------------------------------------- Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:users-subscribe@qpid.apache.org