activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Veloso, Vasco (Coriant - PT/Lisbon)" <Vasco.Vel...@coriant.com>
Subject RE: Expired non-persistent messages on topic appear to not be discarded
Date Thu, 26 Jan 2017 12:45:41 GMT
Hi all,

I finally figured out what was happening. Here are my conclusions, for future reference.

In a nutshell:

* Every time a message expires on the consumer, the prefetch extension kicks in and allows
even more messages to be dispatched. Because the session is not closed, the extended prefetch
limit does not decrease. New expirations will continue to increase it.
* In-flight message size metrics were introduced some time ago. The current implementation
requires all in-flight messages to be kept in memory. So with the prefetch extension growing
uncontrollably, more and more messages are retained in memory. This causes heap exhaustion.
* Messages expired on the broker were counted towards the dispatched count. This was being
hidden by the prefetch extension. When finally it became possible to disable the prefetch
extension, consumers stopped receiving messages after a while.

This resulted in two ActiveMQ code changes:

Expired messages counting as Dispatched on TopicSubscription
https://issues.apache.org/jira/browse/AMQ-6579

TopicSubscription does not honor usePrefetchExtension
https://issues.apache.org/jira/browse/AMQ-6577

Once it became possible to disable prefetch extension for topic subscriptions and the message
counting bug was fixed, the broker became stable.

The only thing I do not understand is the purpose of never decreasing the prefetch extension
value while the session is active...

Regards,

Vasco Veloso

-----Original Message-----
From: Veloso, Vasco (Coriant - PT/Lisbon) [mailto:Vasco.Veloso@coriant.com] 
Sent: sexta-feira, 20 de janeiro de 2017 14:03
To: users@activemq.apache.org
Subject: RE: Expired non-persistent messages on topic appear to not be discarded

All consumers continue to receive messages. 

My first and biggest question is whether this JMX counter evolution makes sense? Is there
any possibility of expired messages counting towards in flight messages?

I am worried that internal data structures may not be released, which will lead either to
an OOM condition or to the JVM experiencing full GCs continuously should the situation continue
for a long time. In fact this has happened during my tests. Heap usage keeps increasing over
several hours, it is then exhausted and the JVM starts to run full GC cycles continuously.

Vasco

-----Original Message-----
From: tbain98@gmail.com [mailto:tbain98@gmail.com] On Behalf Of Tim Bain
Sent: sexta-feira, 20 de janeiro de 2017 13:49
To: ActiveMQ Users <users@activemq.apache.org>
Subject: Re: Expired non-persistent messages on topic appear to not be discarded

Are your slow consumers still receiving messages properly when this happens?  Are there any
indications that this is a "real" problem rather than just a bug in the JMX counters?

Tim

On Jan 19, 2017 8:38 AM, "Veloso, Vasco (Coriant - PT/Lisbon)" < Vasco.Veloso@coriant.com>
wrote:

> Hello,
>
>
>
> I’m using ActiveMQ 5.14.3 to test and try to find a suitable 
> configuration for a scenario where we have multiple producers and 
> multiple consumers of a topic, non-durable, in which some consumers 
> may be quite slow. Also, some messages have TTL defined and others do not.
>
>
>
> By looking at the JMX counters for the topic, InFlightCount remains 
> stable at the prefetch value for consumers. However once messages 
> begin to expire, it looks like they are not discarded:
>
>
>
> ·         ExpiredCount starts to increase as messages time out;
>
> ·         InFlightCount starts to increase at the same rate as
> ExpiredCount.
>
>
>
> JMX counters for slow consumers follow the same pattern. They remain 
> stable until messages begin to expire: MessageCountAwaitingAcknowledge 
> remains near PrefetchSize. When messages expire then 
> MessageCountAwaitingAcknowledge starts to increase as well.
>
>
>
> These counters will never decrease or stabilize as long as there are 
> messages being produced.
>
>
>
> I tried many settings but I still don’t understand what is going on. 
> It appears as if expired messages are forwarded to consumers or expire 
> on consumers? However for this test the broker, producers and 
> consumers are on the same machine so there is no clock drift. And if 
> messages would expire on consumers, wouldn’t they be dropped from the topic as well?
>
>
>
> My broker configuration and test producer/consumer code is attached 
> for your reference.
>
>
>
> I hope that someone can shed some light on this matter, since I can’t 
> make heads or tails of it at this time.
>
>
>
> Thanks!
>
>
>
> Vasco Veloso
>
>
>
Mime
View raw message