activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robin Kåveland Hansen (JIRA) <j...@apache.org>
Subject [jira] [Commented] (AMQ-3966) Messages get stuck in queue
Date Mon, 13 Aug 2012 10:59:38 GMT

    [ https://issues.apache.org/jira/browse/AMQ-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13433059#comment-13433059
] 

Robin Kåveland Hansen commented on AMQ-3966:
--------------------------------------------

This seems to be manifesting in two different ways. I was able to create a situtation in which
a queue reported to having -219 messages. I did this by starting 3 jobs simultaneously that
tried to move the same 400 messages from one queue to another by calling QueueViewMBean.moveMessageTo.
All 400 messages were moved, but afterwards the queue had very different enqueuecount and
dequeuecount. Neither queue had consumers aside from these jobs. Sending messages to the broken
queue resulted in the amount of messages in the queue increasing, but they were not visible
through the use of QueueViewMBean.browse. Using QueueViewMBean.getMessage, I was able to obtain
the messages by JMSMessageID. After sending 13 messages to the queue, I found a log entry
saying toPageIn=13 for this queue.

Thinking I had a workaround for the issue by fetching the message ids from the broker database,
I did a quick experiment with another broken queue. I was not able to extract messages from
this one, using the same method. In this case, both dequeuecount and enqueuecount were 0.
toPageIn=200 for this queue.
                
> Messages get stuck in queue
> ---------------------------
>
>                 Key: AMQ-3966
>                 URL: https://issues.apache.org/jira/browse/AMQ-3966
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.5.0, 5.5.1, 5.6.0
>         Environment: java version "1.6.0_32"
> Java(TM) SE Runtime Environment (build 1.6.0_32-b05)
> Java HotSpot(TM) 64-Bit Server VM (build 20.7-b02, mixed mode)
> Linux ****** 2.6.18-308.4.1.el5 #1 SMP Tue Apr 17 17:08:00 EDT 2012 x86_64 x86_64 x86_64
GNU/Linux
> CentOS release 5.8 (Final)
> Kernel \r on an \m
>            Reporter: Robin Kåveland Hansen
>            Priority: Critical
>         Attachments: brokerlog.zip
>
>
> The broker is using the jdbc-persistence-adapter and persists to an oracle database.
We have so far been unable to reproduce this issue reliably, but it appears both in our testing
and production environments with some regularity. 
> The issue we are having is superficially very similar to [this description|http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-td3244342.html].
Retrieving the queue message count with jmx or inspecting the activemq database with a {code}SELECT
COUNT(*) FROM ACTIVEMQ_MSGS WHERE CONTAINER LIKE '%DeadLetterQueue%'{code} results in a number
that is larger than the amount of messages which can actually be retrieved using a QueueBrowser
or a QueueViewMBean. This difference grows over time, as no messages that are sent to the
queue after this problem occurs are retrievable. We have also observed this problem happening
on queues with normal JMS consumers, where the queue will grow without the consumer receiving
any messages.
> A broker restart can sometimes resolve this issue. Our current workaround is to use a
program that fetches the rows for the queue from the database, unmarshals them and sends them
to a temporary queue. The queue with the stuck messages does not work (does not deliver any
messages to consumers or browsers) until after it has been purged. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message