activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Justin Bertram <jbert...@apache.com>
Subject Re: Consumer throttling
Date Tue, 02 Feb 2016 13:59:24 GMT
For what it's worth, I believe the ActiveMQ Artemis broker (a sub-project of ActiveMQ) has
the feature you're looking for.  See more details in the documentation [1].


Justin

[1] http://activemq.apache.org/artemis/docs/1.2.0/flow-control.html

----- Original Message -----
From: "Bilgin Ibryam" <bibryam@gmail.com>
To: users@activemq.apache.org
Sent: Tuesday, February 2, 2016 5:54:28 AM
Subject: Re: Consumer throttling

Thanks all for your inputs. It seems there is no solution for my
question in ActiveMQ yet.

@Christopher: I'm aware of scheduling feature, but that is more of a
Delayer rather than Throttler. It can delay messages, but cannot
ensure throttled consumption. If I get a burst of messages delay them,
at a later point in time there still will be a burst of messages.
Nothing has improved.

@Tim: I have described your suggestion with two queues and Camel in
between in as a temporary solution in my recent Camel Design Patterns
book, but that solution is not ideal either. It is still possible to
have messages accumulate in queueu B (for example if the consumers are
stopped temporarily) and then have non throttled consumption breaking
SLAs, etc.

@Rau I thoughtl PFC should take care of the message backlog and
protect the broker regardless of the reason for message accumulation,
whether that is slow consumers, throttled consumption or consumers
being down for some reason. Having prefetch limit of 1 is good
suggestion though, thanks.



On 30 January 2016 at 21:30, Tim Bain <tbain@alumni.duke.edu> wrote:
> You could achieve your goal by having two queues, A and B.  Have producers
> publish to A and consumers consume from B, and use an embedded Camel route
> to move messages from A to B at the desired rate.
>
> But Raul's right: if you're going to do this, make sure there's some kind
> of safety feature to prevent messages from backlogging on the broker.
> Maybe you get it for free based on the characteristics of your clients, or
> maybe that's some mechanism to discard built-up messages (based on time,
> count, bytes, etc.), or a way to detect a backlog and temporarily speed up
> the Camel route or to let consumers consume directly from A for a while,
> or...
>
> Tim
> On Jan 30, 2016 12:56 PM, "Raul Kripalani" <raulk@apache.org> wrote:
>
>> Just one remark. This could be dangerous as it could lead to an
>> ever-growing backlog of messages in AMQ. I suppose you have already
>> considered it, and either the producers are slower than the consumers, or
>> you have multiple competing consumers, or there is a period in the day
>> where the producers stop producing to give the consumers time to consume
>> the existing backlog or something.
>>
>> If you implement this, I would reduce the prefect buffer to 1 if you have
>> multiple competing consumers to achieve decent distribution and avoid
>> starvation.
>>
>> Raúl.
>> On 30 Jan 2016 18:15, "Bilgin Ibryam" <bibryam@gmail.com> wrote:
>>
>> > Hi all,
>> >
>> > Apologoes if this is a stupid question. But I wonder is there a way to
>> > do throttling on the consumer side. Let me clarify what I mean.
>> >
>> > PFC can do throttling for the producer, but I need a similar behaviour
>> > on the consumer. For example I want to configure a message consure or
>> > message queue so that messages are consumed/released one per second.
>> > Usually  the messages are consumed as fast as the cosnumers can take.
>> > Then I can do some throttling in consumer Camel route, but that
>> > throutling in Camel cannot be clustered accross instances. If the
>> > throttling can be configured ot queue level, then I don't have to do
>> > consumer throttling.
>> >
>> > Thanks,
>> >
>> > --
>> > Bilgin Ibryam
>> > Camel Committer at ASF & Integration Architect at Red Hat
>> > Blog: http://ofbizian.com | Twitter: @bibryam
>> >
>> > Camel Design Patterns https://leanpub.com/camel-design-patterns
>> > Instant Apache Camel Message Routing http://www.amazon.com/dp/1783283475
>> >
>>



-- 
Bilgin Ibryam
Camel Committer at ASF & Integration Architect at Red Hat
Blog: http://ofbizian.com | Twitter: @bibryam

Camel Design Patterns https://leanpub.com/camel-design-patterns
Instant Apache Camel Message Routing http://www.amazon.com/dp/1783283475

Mime
View raw message