Mailing-List: contact dev-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@activemq.apache.org
Date: Thu, 19 Jul 2012 11:50:35 +0000 (UTC)
From: "Matan Zruya (JIRA)" <jira@apache.org>
To: dev@activemq.apache.org
Message-ID: <213266981.75335.1342698635322.JavaMail.jiratomcat@issues-vm>
In-Reply-To: 
 <1279962701.21149.1330275173300.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (AMQ-3739) ActiveMQ broker stops functioning
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/AMQ-3739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418239#comment-13418239 ] 

Matan Zruya commented on AMQ-3739:
----------------------------------

watch my reply above, 
after removing the DMLC component that consumes messages from activemq and implementing my own solution, the problem described above does not persist anymore.

                
> ActiveMQ broker stops functioning
> ---------------------------------
>
>                 Key: AMQ-3739
>                 URL: https://issues.apache.org/jira/browse/AMQ-3739
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker, Performance Test
>    Affects Versions: 5.4.2, 5.5.0, 5.6.0
>         Environment: Windows server 2008
> 12GB Ram
> 12 cores
> spring integration
> embedded activemq network of brokers
>            Reporter: Matan Zruya
>            Priority: Critical
>
> i'm encountering critical problems with activemq in our dev enviroment,
> activemq brokers seem to hang and not respond to other brokers or jms consumers/producers.
> my example setup is an activemq network of embedded brokers (2 to be precise), 
> one process with an embedded broker produces work, 
> a worker process with an embedded broker as well that does the work.
> 2 queues, a running-queue and a stopped-queue, each msg has a counter which is decremented every time work is done on it until 0 and then the msg is passed to the stopped-queue, where it is consumed by the work producer.
> the logic is implemented with basic spring integration.
> activemq is configured with plenty of memory to spare, and no persistency, jmx is exposed to allow monitoring of the queues,
> what happens is that occasionally msg's get stuck inflight, meaning they do not get an ack, sometimes the inflight is between brokers, and sometimes between an embedded broker and it's internal jvm consumer.
> the worker broker seems to hang, the jmx statistics seem to be frozen, although they do refresh, the broker responds to force restarts of the network connector, and sometime the restart frees msg's until it gets stuck again.
> if i leave the worker on after it's stuck and relaunch the producer, both brokers don't seem to communicate, the produced msg's on one side, don't reach the other, same goes for all advisory msg's related stats (such as producer/consumer counts)
> i tried changing every bit of configuration, with the broker, and with spring itself, but i have a feeling the problem doesn't have anything to do with spring (because of inner broker communication issues), the problem is easily reproducible on a powerful machine where the msg flow is fast, adding persistency or doing it on a normal desktop slows everything down and makes it more difficult to reproduce,
> i tried all recent activemq versions up to the latest 5.6 snapshot.
> i attached the relevent junit tests, one needs to launch the agentTest, and then the producerTest,
> my recommendation is leaving agentTest on and relaunching producerTest over and over again, producerTest should be finished when all it's messages are done (do not launch 2 producer processes at the same time otherwise they will never be finished)
> relevant maven test cases :
> http://db.tt/diVWPNnJ

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira