activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sinu Sekhar (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AMQ-4208) Stuck Message Issue with ActiveMQ 5.6
Date Wed, 05 Dec 2012 23:21:58 GMT
Sinu Sekhar created AMQ-4208:
--------------------------------

             Summary: Stuck Message Issue with ActiveMQ 5.6
                 Key: AMQ-4208
                 URL: https://issues.apache.org/jira/browse/AMQ-4208
             Project: ActiveMQ
          Issue Type: Bug
          Components: activemq-pool, Broker
    Affects Versions: 5.6.0
         Environment: Production
            Reporter: Sinu Sekhar
            Priority: Critical


We have been facing a stuck message issue with 5.6 for a while now. This week I got a chance
to reproduce this in a non-production environment every time I ran a high load.

We use Spring DMLC with the following configuration.
    <bean id="asyncServiceContainer2" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
        <property name="connectionFactory" ref="asyncServiceConnectionFactory2" />
        <property name="destination" ref="asyncServiceDestination" />
        <property name="messageListener" ref="asyncService" />
        <!-- Cache JMS connection -->
        <property name="cacheLevelName" value="CACHE_CONNECTION" />
        <!-- The maximum number of concurrent consumers to create -->
        <property name="maxConcurrentConsumers" value="30" />
        <property name="receiveTimeout" value="10000" />
        <property name="maxMessagesPerTask" value="20" />
        <property name="idleTaskExecutionLimit" value="5" />
    </bean>


The broker uses the default settings with producerFlowControl on.

If I send a jmeter load with 100 concurrent users, with 1000 requests from each user...it
generates a throughput of around 60TPS/node. This is for the producer side. 

I see no issues from a producer standpoint. 

But everytime I run this test, it reaches a point where the deque count just freezes up and
the enqueue keeps incrementing because the producer is not affected. This freeze affects all
consumers. The only way to unlock it is by restarting the broker. I have seen it being released
by restarting the app as well.

When this free happens, there is usually a large number of *.log files in the kahadb folder
and the size of the b-tree file db.data also keeps incrementing indicating the backlog. 

I have taken jstack traces for the broker as well as the application, which I will attach
with this request. 

Please note that this happens everytime I run with the setup shown.

One way I can prevent the issue from happening so frequently, is by turning off dynamic scaling
by using concurrentConsumers to a high number and setting maxMessagesPerTask to the default
(-1 or infinity) for Spring DMLC. This keeps the number of consumers constant and increased
the consumption rate, preventing the backlog from happening. In addition to these changes
I also had to switch from CACHE_CONNECTION cache type to CACHE_CONSUMER to prevent this from
happening.

I have seen a similar issue reported for an earlier version at http://mail-archives.apache.org/mod_mbox/activemq-users/201005.mbox/%3Cg2v4e6f67721005071419u5ca08751h569775ee4e682c5e@mail.gmail.com%3E




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message