activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From drsnyder <s...@dsnyder.com>
Subject ActiveMQ stuck messages when 'redelivered = true'
Date Thu, 09 Jul 2009 16:44:48 GMT

I am seeing some strange behavior from activemq 5.2.0 related to delivery of
messages with a redelivered flag.

We have configured the broker to use jdbc persistence (no journal) talking
to mysql and are using the STOMP protocol.  For testing, I have a consumer
that  for each message will BEGIN a transaction, ACK message on the first
queue , and SEND a copy of the message (being careful to clear out the
appropriate headers like message-id) to a different queue.  It then waits
for a RECEIPT message and that triggers a COMMIT.

If I put persistent messages in the queue, start my consumer but kill it
before exhausting the messages, and restart I always seem to end up with a
few messages that are “stuck”  I can browse the messages (they show
“redelivered=true”) in the admin console, but nothing short of restarting
the broker will allow them to be redelivered.

Upon restarting the consumer, it appears that activemq attempts to push the
message (see log output below) but the stomp client never sees them, and
that activity never gets logged in the stomp trace.

Any ideas what could be happening and if there is a workaround?  I have
thread dumps as well, but jconsole does not detect any deadlocks.  On last
data point, if I put new messages in the queue they get processed as I would
expect but the stuck messages remain.

I really want to avoid regular restarts of activemq in production, so any
help would be greatly appreciated!
David Snyder 


-------------------- QUEUE STATS --------------------

Queue->orders->Attributes:

DequeueCount: 996
DispatchCount: 996
EnqueueCount: 1000
InFlightCount: 114
QueueSize: 4
ProducerFlowControl: false
UseCache: true

Note:  I had a pretty high prefetch for this test, and I think the in flight
count reflects the number of messages that were delivered, but never ack’d
due to killing the consumer.  I see the same behavior with a prefetch of 1.
-------------------- ACTIVEMQ LOG --------------------


DEBUG Queue                          - queue://orders - Recovery - Message
pushed '12446788 - Message ID:dsynder-1761-1247148825394-1:0:-1:1:1
dropped=false locked=true' to subscription: 'QueueSubscription:
consumer=ID:dsynder-1761-1247148825394-1:3:-1:4, destinations=1,
dispatched=0, delivered=0, pending=0'
DEBUG Queue                          - queue://orders - Recovery - Message
pushed '17435985 - Message ID:dsynder-1761-1247148825394-1:0:-1:1:2
dropped=false locked=true' to subscription: 'QueueSubscription:
consumer=ID:dsynder-1761-1247148825394-1:3:-1:4, destinations=1,
dispatched=0, delivered=0, pending=0'
DEBUG Queue                          - queue://orders - Recovery - Message
pushed '7846644 - Message ID:dsynder-1761-1247148825394-1:0:-1:1:3
dropped=false locked=true' to subscription: 'QueueSubscription:
consumer=ID:dsynder-1761-1247148825394-1:3:-1:4, destinations=1,
dispatched=0, delivered=0, pending=0'
DEBUG Queue                          - queue://orders - Recovery - Message
pushed '27310413 - Message ID:dsynder-1761-1247148825394-1:0:-1:1:4
dropped=false locked=true' to subscription: 'QueueSubscription:
consumer=ID:dsynder-1761-1247148825394-1:3:-1:4, destinations=1,
dispatched=0, delivered=0, pending=0'
DEBUG JDBCPersistenceAdapter         - Cleaning up old messages.DEBUG
DefaultJDBCAdapter             - Executing SQL: DELETE FROM
ACTIVEMQ_MSGSWHERE ( EXPIRATION<>0 AND EXPIRATION<?) OR ID <= ( SELECT
min(ACTIVEMQ_ACKS.LAST_ACKED_ID) FROM ACTIVEMQ_ACKS WHERE
ACTIVEMQ_ACKS.CONTAINER=ACTIVEMQ_MSGS.CONTAI
NER)
DEBUG DefaultJDBCAdapter             - Deleted 0 old message(s).
DEBUG JDBCPersistenceAdapter         - Cleanup done.


-------------------- STOMP TRACE --------------------

2009-07-09 10:42:07,579 [0.240.145:50282] TRACE StompTransportFilter          
- Received:
SUBSCRIBE
ack:client
activemq.prefetchSize:1
activemq.dispatchAsync:true
destination:/queue/orders
id:34b52eb2560982879bd8febb4590c7b2
x-client:127.0.0.1
x-digest:MD5:d41d8cd98f00b204e9800998ecf8427e

-- 
View this message in context: http://www.nabble.com/ActiveMQ-stuck-messages-when-%27redelivered-%3D-true%27-tp24413676p24413676.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message