activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maurits Lucas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (AMQ-2009) Problem with message dispatch after a while
Date Mon, 15 Dec 2008 10:29:05 GMT

    [ https://issues.apache.org/activemq/browse/AMQ-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=48177#action_48177
] 

Maurits Lucas commented on AMQ-2009:
------------------------------------

We have been able to reproduce this issue in our test runs every time.

The only problem, and the reason I haven't got a unit test for you, is that it takes a very
long time to rear its ugly head.

The minimum setup which exposes the issue is have a producer (P1) send a lot (6 million in
our case) messages to a topic (t1) and have one consumer (C1) consume the messages from the
topic and publish them to a queue (q1).

P1 --> [t1] --> C1 --> [q1] 

Both P1 and C1 produce and consume / produce messages as fast as they can. Because P1 produces
faster than C1 can consume, we get a fast producer / slow consumer for that topic.

At the end of the run, there are 6 million messages on the queue, but if you add a consumer
to the queue, it won't consume any messages. If you browse the queue using the webconsole,
it appears empty. Only restarting ActiveMQ gets q1 working again.

In our previous tests, we discovered that this "freezing" of dispatching (it affects other
queues than q1 as well) starts somewhere during the publishing of the 6 million messages to
t1.

We are using a standalone broker, Kaha for persistence and Spring 2.5.4 for JMS support.

OS: Linux (2.6.18-92.1.18.el5 #1 SMP Wed Nov 12 09:19:49 EST 2008 x86_64 x86_64 x86_64 GNU/Linux)
Java: 
  java version "1.6.0_10"
  Java(TM) SE Runtime Environment (build 1.6.0_10-b33)
  Java HotSpot(TM) 64-Bit Server VM (build 11.0-b15, mixed mode)

We have tested with 5.1.0, 5.2.0 and 5.3-SNAPSHOT. Using 5.1.0 and the snapshot we ran into
the issue, with 5.2.0 we ran into AMQ-2020 and so didn't test any further.

We saw some peculiar values for the inflight counter on t1, this leveled out at 32766 for
hours on end, sometimes going down to 32765 for a couple of seconds. Not sure if that is significant,
but I am including a screenshot of JConsole just in case.


> Problem with message dispatch after a while
> -------------------------------------------
>
>                 Key: AMQ-2009
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2009
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.1.0, 5.2.0
>            Reporter: Rajani Chennamaneni
>            Priority: Blocker
>         Attachments: DispatchMultipleConsumersTest.java
>
>
> Messages are not getting dispatched after a while (although it accepts new incoming messages)
until restart of the broker. This problem is described in several posts.
> http://www.nabble.com/Pending-Messages-are-shown-in-ActiveMQ-td20241332.html
> http://www.nabble.com/Consumer-Listener-stop-receving-message-until-ActiveMQ-restart-td20355247.html
> http://www.nabble.com/Stuck-messages---Dispatch-issues-td20467949.html
> There was also an issue opened in Spring project for this thinking it was Spring problem.
> http://jira.springframework.org/browse/SPR-5110
> I am not able to reproduce with Junit test case having BrokerService started with in
the test case. I guess I am not hitting the right stress conditions this way. But when I run
the test case against an externally running ActiveMQ instance backed with oracle database
persistence, it is reproducible most of the times. This is not a every time failure situation,
it takes more time once than the other.
> I was able to hit this situation of stuck messages on queue using following scenario
most of the times:
> 1) Start 2 concurrent consumers for the queue using Spring's DefaultMessageListenerContainer
using cacheLevelName as CACHE_CONSUMER
> 2) Send messages using JMETER 2.3.2 to the queue on ActiveMQ stand alone broker instance
with 50 threads looping 20 times.
> 3) After a while, you will notice that Spring logs that no messages are being received
but the messages are shown jconsole of ActiveMQ and the database backing it for persistence.
> But in 5.2 RC3, the problem is that it dispatches duplicate messages and does not remove
them from broker's database after acknowledge properly.
> Attached test case might help to reproduce when run against externally running stand
alone ActiveMQ broker. Another way to see the problem is that try to load test using JMETER
by sending messages to a queue with a camel route that moves messages from this queue to another
and you will notice that it stops moving after while or copied duplicates in case of 5.2 RC3.
> Sorry about such a huge description but it is a real problem! A different team at our
company are having this issue in production with 5.1. They are using it as an embedded broker
with derby for persistence.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message