activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ying (JIRA)" <j...@apache.org>
Subject [jira] Reopened: (AMQ-2009) Problem with message dispatch after a while
Date Wed, 10 Jun 2009 21:45:35 GMT

     [ https://issues.apache.org/activemq/browse/AMQ-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

ying reopened AMQ-2009:
-----------------------


I see this in Trunk 771718. Our network of 4 brokers are running for almost 30 days, then
we see a broker is not dispatching msgs. It has about 400mb data in its data directory. 

We restart the application to talk to another broker, restart the problematic broker, it starts
fine, bridged ok, sees the consumer on the other broker, but all the msgs stuck on this broker
is not dispatched. They are stuck.

I will try to gather more information when I get more idea. A simple unit test of the case
is ok with less msgs. I suspect this happens when a lot of msgs are stuck on the queue, then
it fails to dispatch even after restart. BTW, we have enough memory on the box for the broker
to run and jconsole show it is using only a portion. There is no error in the log with debug
turned on.

DemandForwardingBridgeSupport's serviceLocalCommand is supposed to be called but not. Any
possible threading issue? Regarding the demandforwarding bridge, do you know the name of the
thread I shall look in jconsole. due to the complexity of our system, there are about 170
live threads in the jconsole for this broker. Maybe a thread is blocked. 

Any suggestion is welcome. I am looking into this issue until i find a fix. 

> Problem with message dispatch after a while
> -------------------------------------------
>
>                 Key: AMQ-2009
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2009
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.1.0, 5.2.0
>            Reporter: Rajani Chennamaneni
>            Assignee: Rob Davies
>            Priority: Blocker
>             Fix For: 5.3.0
>
>         Attachments: AMQ-2009Testcase2.zip, consumertest.zip, DispatchMultipleConsumersTest.java,
JConsole-screenshot.jpg, testcase.zip
>
>
> Messages are not getting dispatched after a while (although it accepts new incoming messages)
until restart of the broker. This problem is described in several posts.
> http://www.nabble.com/Pending-Messages-are-shown-in-ActiveMQ-td20241332.html
> http://www.nabble.com/Consumer-Listener-stop-receving-message-until-ActiveMQ-restart-td20355247.html
> http://www.nabble.com/Stuck-messages---Dispatch-issues-td20467949.html
> There was also an issue opened in Spring project for this thinking it was Spring problem.
> http://jira.springframework.org/browse/SPR-5110
> I am not able to reproduce with Junit test case having BrokerService started with in
the test case. I guess I am not hitting the right stress conditions this way. But when I run
the test case against an externally running ActiveMQ instance backed with oracle database
persistence, it is reproducible most of the times. This is not a every time failure situation,
it takes more time once than the other.
> I was able to hit this situation of stuck messages on queue using following scenario
most of the times:
> 1) Start 2 concurrent consumers for the queue using Spring's DefaultMessageListenerContainer
using cacheLevelName as CACHE_CONSUMER
> 2) Send messages using JMETER 2.3.2 to the queue on ActiveMQ stand alone broker instance
with 50 threads looping 20 times.
> 3) After a while, you will notice that Spring logs that no messages are being received
but the messages are shown jconsole of ActiveMQ and the database backing it for persistence.
> But in 5.2 RC3, the problem is that it dispatches duplicate messages and does not remove
them from broker's database after acknowledge properly.
> Attached test case might help to reproduce when run against externally running stand
alone ActiveMQ broker. Another way to see the problem is that try to load test using JMETER
by sending messages to a queue with a camel route that moves messages from this queue to another
and you will notice that it stops moving after while or copied duplicates in case of 5.2 RC3.
> Sorry about such a huge description but it is a real problem! A different team at our
company are having this issue in production with 5.1. They are using it as an embedded broker
with derby for persistence.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message