Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Message-ID: <4B5770C1.8050807@tweakers.net>
Date: Wed, 20 Jan 2010 22:08:17 +0100
From: Arjen van der Meijden <acmmailing@tweakers.net>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US;
 rv:1.9.1.5) Gecko/20091204 Lightning/1.0b1 Thunderbird/3.0
MIME-Version: 1.0
To: users@activemq.apache.org
Subject: Gradually increasing, huge cpu-load when using Stomp+nio
Content-Type: multipart/mixed;
 boundary="------------060701030607080001000407"

--------------060701030607080001000407
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi List,

I just restarted a activemq 5.3-instance which was causing 1000% cpu 
(the machine has 8 cores, so all where fully working) and a load of 1035.

I'm pretty confident that this load/cpu-usage was caused by using 
stomp+nio rather than 'stomp' (whether it was just nio or not, I don't 
know).

Our usage-pattern is that we have a few queues with a single consumer 
each. The producers are php-web-processes that make a connection using 
stomp, send a few messages to some of the queues and disconnect again 
(which is why we have the transport.closeAsync=false). The heaviest two 
queues receive about 30-50 messages/second during peak hours (and thus, 
ActiveMQ receives about 30-50 connections/second).

We also have a few topics that get messages quite infrequently, from 
similar stomp-producers, but those are consumed with java-based 
applications.

The load/cpu-usage apparently has gradually increased, as can be seen in 
these two graphs:
Cpu-usage
http://tweakers.net/stats/?Action=Generator&Mode=Serverstats&Time=1263919627&Dagen=1&StatsServer=Argus&colServers=CPUUsage

Load average
http://tweakers.net/stats/?Action=Generator&Mode=Serverstats&Time=1263919627&Dagen=1&StatsServer=Argus&colServers=LoadAvg

The increase in load has gone steadily in just a few hours up untill the 
moment where it took the php-processes more than 13 seconds to actually 
connect, send two messages and disconnect (rather than a few milliseconds).

Our activemq.xml is attached, although I already replaced stomp+nio with 
normal stomp. Our ACTIVEM_OPTS is "-Xms2048M -Xmx2048M 
-XX:+UseParallelOldGC -server 
-Dorg.apache.activemq.UseDedicatedTaskRunner=true" (i.e. the default 
with adjusted memory and GC)

The increasing load hasn't appeared yet with the adjusted config that 
uses non-nio connections.

Unfortunately, this is a production server so I can't run tests very 
easily. Nor do I have a problem with not running nio, so this is mainly 
a mail to let you know of this problem, rather than me having a big 
issue with activemq right now :)

Best regards,

Arjen

--------------060701030607080001000407
Content-Type: text/xml;
 name="activemq.xml"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="activemq.xml"

<!--
    Licensed to the Apache Software Foundation (ASF) under one or more
    contributor license agreements.  See the NOTICE file distributed with
    this work for additional information regarding copyright ownership.
    The ASF licenses this file to You under the Apache License, Version 2.0
    (the "License"); you may not use this file except in compliance with
    the License.  You may obtain a copy of the License at
   
    http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
-->
<beans
  xmlns="http://www.springframework.org/schema/beans"
  xmlns:amq="http://activemq.apache.org/schema/core"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
  http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">

    <!-- Allows us to use system properties as variables in this configuration file -->
    <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="locations">
            <value>file:${activemq.base}/conf/credentials.properties</value>
        </property>      
    </bean>

    <!-- 
        The <broker> element is used to configure the ActiveMQ broker. 
    -->
    <broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.base}/data">
        
        <!-- 
            The managementContext is used to configure how ActiveMQ is exposed in 
            JMX. By default, ActiveMQ uses the MBean server that is started by 
            the JVM. For more information, see: 
            
            http://activemq.apache.org/jmx.html 
        -->
        <managementContext>
            <managementContext createConnector="false"/>
        </managementContext>

        <!-- 
            Configure message persistence for the broker. The default persistence
            mechanism is the KahaDB store (identified by the kahaDB tag). 
            For more information, see: 
            
            http://activemq.apache.org/persistence.html 
        -->
        <persistenceAdapter>
            <kahaDB directory="${activemq.base}/data/kahadb"/>
        </persistenceAdapter>
        
        
        <!--
			For better performances use VM cursor and small memory limit.
			For more information, see:
            
            http://activemq.apache.org/message-cursors.html
            
            Also, if your producer is "hanging", it's probably due to producer flow control.
            For more information, see:
            http://activemq.apache.org/producer-flow-control.html
        -->
              
<!--        <destinationPolicy>
           <policyMap>
              <policyEntries>
                <policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
                  <pendingSubscriberPolicy>
                    <vmCursor />
                  </pendingSubscriberPolicy>
                </policyEntry>
                <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">
                  <!- - Use VM cursor for better latency
                       For more information, see:
                       
                       http://activemq.apache.org/message-cursors.html
                       
                  <pendingQueuePolicy>
                    <vmQueueCursor/>
                  </pendingQueuePolicy>
                  - ->
<!- -                </policyEntry>
              </policyEntries>
            </policyMap>
        </destinationPolicy> -->
 
         <!--
            The systemUsage controls the maximum amount of space the broker will 
            use before slowing down producers. For more information, see:
            
            http://activemq.apache.org/producer-flow-control.html
             
        <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage limit="20 mb"/>
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="1 gb" name="foo"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="100 mb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>
		-->
		  
        <!-- 
            The transport connectors expose ActiveMQ over a given protocol to
            clients and other brokers. For more information, see: 
            
            http://activemq.apache.org/configuring-transports.html 
        -->
        <transportConnectors>
            <transportConnector name="openwire" uri="tcp://0.0.0.0:61616?transport.closeAsync=false"/>
            <transportConnector name="stomp" uri="stomp://0.0.0.0:61613?transport.closeAsync=false" />
        </transportConnectors>

    </broker>

    <!-- 
        Uncomment to enable Camel
        Take a look at activemq-camel.xml for more details
         
    <import resource="camel.xml"/>
    -->

    <!-- 
        Enable web consoles, REST and Ajax APIs and demos
        Take a look at activemq-jetty.xml for more details 
    -->
    <import resource="jetty.xml"/>
    
</beans>

--------------060701030607080001000407--