activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sree Panchajanyam D (JIRA)" <>
Subject [jira] [Commented] (AMQ-3460) Messages get stuck in the queue for some client sessions, "Dispatched Queue" grows but "Dequeues" doesn't move
Date Wed, 06 Nov 2013 17:27:17 GMT


Sree Panchajanyam D commented on AMQ-3460:

Root Cause Analysis (Saving the devil for the very last)
This issue falls in no mans land between spring code and activemq code .

In my example above the main thread of spring tries to connect for first 60 seconds (first
timeout), when it fails to connect to ActiveMQ as it is not up, leaves it to listener threads
( if there are 20 concurrentConsumers we would have 20 DMLC threads) to recover. The main
thread of spring does not clear the connection object of spring and hands over a stale connection
reference for the listeners to work with.
If ActiveMQ starts after the first time out and before the second timeout one of the 20 listener
threads creates sessions on an unstarted connection. When ActiveMQ delivers a message to the
consumer, the message flow into the consumer JVM/process but DMLC threads cannot receive them
and hence the issue.When this issue occurs you can notice the following warning in the logs

"trying to receive messages on an unstarted connection" and subsequent messages are not delivered".

This warning is logged for the first message alone and not for the subsequent messages.

We found easiest and logical of all places to fix the issue. 

> Messages get stuck in the queue for some client sessions, "Dispatched Queue" grows but
"Dequeues" doesn't move
> --------------------------------------------------------------------------------------------------------------
>                 Key: AMQ-3460
>                 URL:
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.3.1
>         Environment: java version "1.5.0_15"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_15-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_15-b04, mixed mode)
>            Reporter: Timur Evdokimov
>            Priority: Critical
>             Fix For: AGING_TO_DIE
>         Attachments: acmq01.png, acmq02.png
> We experience the following problem.
> There's 1 sender thread and 20 listener threads operating on the same queue.
> No other queues are involved.
> Message traffic goes generally OK via the queue, however Number of pending messages goes
up one by one.
> Looking at the active customers list, the following appears (see attached screenshot).
> For all sessions but two, the messages are dispatched properly.
> For these two affected sessions, the "Dispatched Queue" value gots stuck at certain moment
and doesn't deliver anymore.
> Eventually, more sessions got affected, and maybe after some hours, the stuck messages
got delivered.
> What we tried - adding useCache="false" into <policyEntry queue=">" ... > tag.
Didn't have any effect.

This message was sent by Atlassian JIRA

View raw message