activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matan Zruya (Created) (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AMQ-3739) ActiveMQ broker stops functioning
Date Sun, 26 Feb 2012 16:52:53 GMT
ActiveMQ broker stops functioning
---------------------------------

                 Key: AMQ-3739
                 URL: https://issues.apache.org/jira/browse/AMQ-3739
             Project: ActiveMQ
          Issue Type: Bug
          Components: Broker, Performance Test
    Affects Versions: 5.5.0, 5.4.2, 5.6.0
         Environment: Windows server 2008
12GB Ram
12 cores
spring integration
embedded activemq network of brokers
            Reporter: Matan Zruya
            Priority: Critical


i'm encountering critical problems with activemq in our dev enviroment,
activemq brokers seem to hang and not respond to other brokers or jms consumers/producers.


my example setup is an activemq network of embedded brokers (2 to be precise), 
one process with an embedded broker produces work, 
a worker process with an embedded broker as well that does the work.

2 queues, a running-queue and a stopped-queue, each msg has a counter which is decremented
every time work is done on it until 0 and then the msg is passed to the stopped-queue, where
it is consumed by the work producer.

the logic is implemented with basic spring integration.

activemq is configured with plenty of memory to spare, and no persistency, jmx is exposed
to allow monitoring of the queues,
what happens is that occasionally msg's get stuck inflight, meaning they do not get an ack,
sometimes the inflight is between brokers, and sometimes between an embedded broker and it's
internal jvm consumer.

the worker broker seems to hang, the jmx statistics seem to be frozen, although they do refresh,
the broker responds to force restarts of the network connector, and sometime the restart frees
msg's until it gets stuck again.
if i leave the worker on after it's stuck and relaunch the producer, both brokers don't seem
to communicate, the produced msg's on one side, don't reach the other, same goes for all advisory
msg's related stats (such as producer/consumer counts)

i tried changing every bit of configuration, with the broker, and with spring itself, but
i have a feeling the problem doesn't have anything to do with spring (because of inner broker
communication issues), the problem is easily reproducible on a powerful machine where the
msg flow is fast, adding persistency or doing it on a normal desktop slows everything down
and makes it more difficult to reproduce,

i tried all recent activemq versions up to the latest 5.6 snapshot.

i attached the relevent junit tests, one needs to launch the agentTest, and then the producerTest,
my recommendation is leaving agentTest on and relaunching producerTest over and over again,
producerTest should be finished when all it's messages are done (do not launch 2 producer
processes at the same time otherwise they will never be finished)

relevant maven test cases :
http://db.tt/diVWPNnJ


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message