activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From benj <apache....@benandi.com>
Subject Re: ActiveMQ slow consumer policy
Date Wed, 19 Dec 2012 14:19:14 GMT
Hi

Sorry - I should have said that I did try preFetch="1", and I also updated
the constantPendingMessage limit to 0.

Here's my config:
                <policyEntry topic=">" topicPrefetch="1"
alwaysRetroactive="true">
                    <subscriptionRecoveryPolicy>
                        <lastImageSubscriptionRecoveryPolicy/>
                    </subscriptionRecoveryPolicy>
                    <pendingMessageLimitStrategy>
                        <constantPendingMessageLimitStrategy limit="0"/>
                    </pendingMessageLimitStrategy>
                    <messageEvictionStrategy>
                        <oldestMessageEvictionStrategy/>
                    </messageEvictionStrategy>
                </policyEntry>

But between the producer and consumer you can see a queue building up, and
the latency increasing second by second:

Producer, sending approx 5000 msgs/s
  Sending message: 3944, 3944 msg/s
  Sending message: 8302, 4358 msg/s
  Sending message: 13471, 5169 msg/s
  Sending message: 18763, 5292 msg/s
  Sending message: 24212, 5449 msg/s
  Sending message: 29331, 5119 msg/s
  Sending message: 35003, 5672 msg/s
  Sending message: 40666, 5663 msg/s
  Sending message: 46207, 5541 msg/s
  Sending message: 51914, 5707 msg/s

Consumer, receiving only 500-1000 msgs/s, latency increasing, no messages
being discarded
  Received message: 1525, 1526 msgs/s, latency 21 < 197 < 439
  Received message: 3289, 1764 msgs/s, latency 350 < 624 < 937
  Received message: 4586, 1297 msgs/s, latency 948 < 1240 < 1595
  Received message: 6390, 1804 msgs/s, latency 1591 < 1983 < 2239
  Received message: 7947, 1557 msgs/s, latency 2235 < 2485 < 2784
  Received message: 8958, 1011 msgs/s, latency 3102 < 3345 < 3625
  Received message: 9464, 506 msgs/s, latency 4422 < 4451 < 4503
  Received message: 9970, 506 msgs/s, latency 5014 < 5061 < 5121
  Received message: 10470, 500 msgs/s, latency 5751 < 5812 < 5873
  Received message: 10971, 501 msgs/s, latency 6409 < 6466 < 6527

Each row is a 1 second sampling period
Messages are simple incrementing ids
Latency is given as min < average < max, in milliseconds

So, I'm kind of at the end of things to try and get this working... but I
hope there's something simple I missing

Thanks again

Ben

From: gtully [via ActiveMQ]
[mailto:ml-node+s2283324n4660890h15@n4.nabble.com] 
Sent: 19 December 2012 13:34
To: benj
Subject: Re: ActiveMQ slow consumer policy

> 1000 topics with a prefetch of 100 is 100,000 pending messages in memory. 
>
> To get real eager discarding try: 
> 
> <policyEntry topic=">" topicPrefetch="1" > 
>                     <subscriptionRecoveryPolicy> 
>                         <lastImageSubscriptionRecoveryPolicy/> 
>                     </subscriptionRecoveryPolicy> 
>                     <pendingMessageLimitStrategy> 
>                         <constantPendingMessageLimitStrategy limit="0"/> 
>                     </pendingMessageLimitStrategy> 
> ... 
> 
> so that it won't keep any messages pending when the consumer is not ready 
> to consume. this will reduce the dispatch work load on the topic and
> ensure 
> fast consumer get all messages while slow consumers get gaps. 
> 
> The lastImageSubscriptionRecoveryPolicy is only for retroactive 
> returning/new consumers, the get the current last state when the connect. 
> 




--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-slow-consumer-policy-tp4660859p4660900.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message