activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joerg Henne <henn...@apache.org>
Subject Strange delivery behaviour with 5.2.0 ResourceAdapter
Date Thu, 30 Jul 2009 12:17:13 GMT

Hi,

we have the following scenario:

- ActiveMQ 5.2.0 is run in embedded fashion inside JBoss using the ActiveMQ
RA
- Messaging is configured to be persistent using Kaha persistence
- A few SessionBeans send messages asynchronously to a queue using
Connections obtained from the RA
- Every once in a while a TimerBean (well, actually it is a MessageBean
driven by JBoss's Quartz integration, but that should not matter - in any
case the consumption happens inside a JTA transaction) comes alive, fetches
as many messages from the queue as it can get (using synchronous
receive-calls), and processes them. The rationale behind this approach is
that the messages in question contain statistical data which can be
aggregated before sending it off to a database. The aggregation dramatically
reduces the load on the database. 
- The messages are not consumed in any other way, in particular they are not
delivered to an MDB

With this scenario we see two strange behaviours which may or may not be
related:

1. If there are many messages in the queue, one would expect any given
invocation of the consumer bean to receive all messages there are. However,
this is not the case - it only gets a limited number of messages, usually
between 800 and 1200 at a time. The actual number varies but is almost
always a multiple of 100. Increasing the timeout on the receive operation
doesn't change anything.

2. Since the switch to 5.2.0 we frequently ran into limits of the
SystemMemory, seemingly because cursors are kept open, even though there is
no persistent consumer. We turned off producer flow control and configured
the queue to use a FileQueueCursor, but we still see the MemoryPercentUsage
go though the roof (>>100%!) with large numbers of messages in the queue.

We were able to address issue 1 by reducing the prefetch count to 10, but I
still think this may point to a deeper issue. With but a superficial look at
the code, my guess at what happens is something along the lines of:
- Connections are pooled and re-used. Which connection is used to consume
messages is essentially random or at least up to the JCA container's pool
implementation.
- Once a consumption cycle hits the broker, a cursor is opened. This cursor
is not closed upon returning the connection to the RA pool. I can see that
the ActiveMQMessageConsumer sends a "remove" command upon close(), which we
do call, but maybe that's not enough. Furthermore I can see that
ActiveMQConnection.cleanup() does some work in that direction, but strangely
it only causes dispose() to be called on the ActiveMQMessageConsumer (via
ActiveMQSession.dispose()), but this method doesn't send anything to the
broker.
- New messages get pre-fetch delivered to the various connections/consumers
in the pool where they pile up until a consumption run "accedentially"
receives them. 
- Consequently, the open cursors consume memory on the broker, which spells
doom for message producers unless producerFlowControl is turned off (we
learned the hard way...).

On the other hand, there are a few observations which don't quite fit this
explanation, so it may be way off the mark. In particular:
- It didn't help to use a separate connection pool just for message
consumption
- The ConsumerCount info in JMX for the queue drops to zero when no
consumption is active
- Every once in a while (but rarely) the consumer actually receives all
messages there are

The described behaviour is rather annoying albeit no longer fatal to the
application since we turned off producer flow control and use the file queue
cursor. If you need any other information to look into the issue, please let
me know.

Thanks
Joerg
-- 
View this message in context: http://www.nabble.com/Strange-delivery-behaviour-with-5.2.0-ResourceAdapter-tp24737329p24737329.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message