activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sinu Sekhar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQ-4208) Stuck Message Issue with ActiveMQ 5.6
Date Wed, 05 Dec 2012 23:47:58 GMT

    [ https://issues.apache.org/jira/browse/AMQ-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510934#comment-13510934
] 

Sinu Sekhar commented on AMQ-4208:
----------------------------------

Just want to add that I used two machines to reproduce this on a non-prod environment, one
for the broker and one for the app. JMeter was run on a different machine, to send load to
the application which acted like a producer as well as consumer ... all part of the same war
file, implemented using Spring.

                
> Stuck Message Issue with ActiveMQ 5.6
> -------------------------------------
>
>                 Key: AMQ-4208
>                 URL: https://issues.apache.org/jira/browse/AMQ-4208
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: activemq-pool, Broker
>    Affects Versions: 5.6.0
>         Environment: Production
>            Reporter: Sinu Sekhar
>            Priority: Critical
>         Attachments: activemq-orig.11302012.xml, activemqthreads6.txt, threads6.txt
>
>
> We have been facing a stuck message issue with 5.6 for a while now. This week I got a
chance to reproduce this in a non-production environment every time I ran a high load.
> We use Spring DMLC with the following configuration.
>     <bean id="asyncServiceContainer2" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
>         <property name="connectionFactory" ref="asyncServiceConnectionFactory2" />
>         <property name="destination" ref="asyncServiceDestination" />
>         <property name="messageListener" ref="asyncService" />
>         <!-- Cache JMS connection -->
>         <property name="cacheLevelName" value="CACHE_CONNECTION" />
>         <!-- The maximum number of concurrent consumers to create -->
>         <property name="maxConcurrentConsumers" value="30" />
>         <property name="receiveTimeout" value="10000" />
>         <property name="maxMessagesPerTask" value="20" />
>         <property name="idleTaskExecutionLimit" value="5" />
>     </bean>
> The broker uses the default settings with producerFlowControl on.
> If I send a jmeter load with 100 concurrent users, with 1000 requests from each user...it
generates a throughput of around 60TPS/node. This is for the producer side. 
> I see no issues from a producer standpoint. 
> But everytime I run this test, it reaches a point where the deque count just freezes
up and the enqueue keeps incrementing because the producer is not affected. This freeze affects
all consumers. The only way to unlock it is by restarting the broker. I have seen it being
released by restarting the app as well.
> When this free happens, there is usually a large number of *.log files in the kahadb
folder and the size of the b-tree file db.data also keeps incrementing indicating the backlog.

> I have taken jstack traces for the broker as well as the application, which I will attach
with this request. 
> Please note that this happens everytime I run with the setup shown.
> One way I can prevent the issue from happening so frequently, is by turning off dynamic
scaling by using concurrentConsumers to a high number and setting maxMessagesPerTask to the
default (-1 or infinity) for Spring DMLC. This keeps the number of consumers constant and
increased the consumption rate, preventing the backlog from happening. In addition to these
changes I also had to switch from CACHE_CONNECTION cache type to CACHE_CONSUMER to prevent
this from happening.
> I have seen a similar issue reported for an earlier version at http://mail-archives.apache.org/mod_mbox/activemq-users/201005.mbox/%3Cg2v4e6f67721005071419u5ca08751h569775ee4e682c5e@mail.gmail.com%3E

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message