activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From DV <vind...@gmail.com>
Subject Re: High swap usage and network timeouts during heavy publishing
Date Wed, 23 Jan 2013 21:35:53 GMT
Just following up... I wonder if anyone else has experienced high swap
usage and/or network timeouts in ActiveMQ and what steps they've taken
to fix that. Thanks!

- DV

On Sun, Jan 20, 2013 at 5:00 PM, DV <vindimy@gmail.com> wrote:
> I forgot to mention one more thing. This sort of behavior (high swap
> usage and lots of timeouts) used to end with "Too many open files"
> exceptions flooding ActiveMQ's logs, until we upped the ulimit on the
> host from 1024 to max (transport.closeAsync=false did _not_ help).
> Here's a sample log entry:
>
> 2013-01-05 12:16:20,352 | ERROR | Could not accept connection :
> java.io.IOException: Too many open files |
> org.apache.activemq.broker.TransportConnector | ActiveMQ Transport
> Server: nio://0.0.0.0:1102?transport.closeAsync=false
>
> - DV
>
> On Fri, Jan 18, 2013 at 12:46 PM, DV <vindimy@gmail.com> wrote:
>> Hi all,
>>
>> I'd like to post some symptoms we're seeing with our ActiveMQ
>> publishing broker and maybe get some ideas on what we need to tweak.
>>
>> We're using ActiveMQ to push publishing messages to a set of pre-prod
>> and prod consumers. During peak of publishing, ActiveMQ becomes
>> unresponsive (web UI timeout, JMX timeout, process hangs). On the
>> system side, we're seeing swap usage approach 50% of host's physical
>> memory (4GB out of 8GB). In logs there are three types of warnings:
>>
>> Slow KahaDB access:
>> 2013-01-17 23:49:58,616 |  INFO | Slow KahaDB access: Journal append
>> took: 0 ms, Index update took 1991 ms
>> 2013-01-17 23:50:26,649 |  INFO | Slow KahaDB access: Journal read took: 1818 ms
>>
>> Timeout of Zookeeper connection:
>> 2013-01-18 03:24:25,195 |  INFO | Client session timed out, have not
>> heard from server in 14251ms for sessionid 0x23c46ac51160442, closing
>> socket connection and attempting reconnect
>>
>> Timeout of consumer connections:
>> 2013-01-18 04:23:58,674 |  WARN | Transport Connection to:
>> tcp://10.129.18.33:48055 failed:
>> org.apache.activemq.transport.InactivityIOException: Channel was
>> inactive for too (>30000) long: tcp://10.129.18.33:48055
>> 2013-01-18 04:23:58,673 |  WARN | Transport Connection to:
>> tcp://10.14.26.60:48237 failed:
>> org.apache.activemq.transport.InactivityIOException: Channel was
>> inactive for too (>30000) long: tcp://10.14.26.60:48237
>> 2013-01-18 04:23:58,673 |  WARN | Transport Connection to:
>> tcp://10.14.26.61:56382 failed:
>> org.apache.activemq.transport.InactivityIOException: Channel was
>> inactive for too (>30000) long: tcp://10.14.26.61:56382
>>
>> We're running ActiveMQ 5.7.0 on RHEL 5.7 with Java 1.6.0_34. The host
>> has 8GB of RAM and has /data and /logs NFS-mounted. There's about 180
>> queues on the broker, and a publishing event usually goes out to about
>> 3-10 queues.
>>
>> We've already tested the underlying infrastructure (host, network,
>> etc.) as well as tweaked ActiveMQ settings to our best knowledge, yet
>> we continue seeing these issues. Recently, we've enabled PFC (producer
>> flow control), which helped bring down the number of occurrences.
>>
>> Here's the startup command:
>>
>> /usr/java/default/bin/java -DuseProfiler=true
>> -javaagent:/usr/local/appdynamics-agent-3.5.5/javaagent.jar
>> -Dappdynamics.agent.nodeName=hostname.fqdn
>> -Dappdynamics.agent.logs.dir=/logs/appdynamics/ -Xmx4096M -Xms4096M
>> -XX:MaxPermSize=128m
>> -Dorg.apache.activemq.UseDedicatedTaskRunner=false
>> -Dcom.sun.management.jmxremote
>> -Dcom.sun.management.jmxremote.ssl=false
>> -Dcom.sun.management.jmxremote.authenticate=false
>> -Dorg.apache.activemq.store.kahadb.LOG_SLOW_ACCESS_TIME=1500
>> -Djava.util.logging.config.file=logging.properties
>> -Dactivemq.classpath=/apps/apache-activemq/conf
>> -Dactivemq.conf=/apps/apache-activemq/conf -jar
>> /apps/apache-activemq/bin/run.jar start
>>
>> Here are the links to some Appdynamics graphs (a publishing activity
>> has started at 8:30am), as well as our ActiveMQ config and complete
>> console output log.
>>
>> https://dl.dropbox.com/u/2739192/amq/activemq.xml
>> https://dl.dropbox.com/u/2739192/amq/broker.log
>>
>> https://dl.dropbox.com/u/2739192/amq/io.png
>> https://dl.dropbox.com/u/2739192/amq/net_io.png
>> https://dl.dropbox.com/u/2739192/amq/jmx.png
>> https://dl.dropbox.com/u/2739192/amq/jvm.png
>> https://dl.dropbox.com/u/2739192/amq/mem.png
>>
>> Here's the portion of ActiveMQ config for ease of use.
>>
>>
>>     <broker xmlns="http://activemq.apache.org/schema/core"
>>         brokerName="broker-publishing-legacy"
>>         useJmx="true"
>>         dataDirectory="/data/activemq/">
>>
>>         <destinationInterceptors>
>>             <bean xmlns="http://www.springframework.org/schema/beans"
>> id="QueueDestinationInterceptor"
>> class="com.abcde.eps.interceptor.QueueDestinationInterceptor">
>>             </bean>
>>             <virtualDestinationInterceptor>
>>                 <virtualDestinations>
>>                     <virtualTopic name="VirtualTopic.>" prefix="Consumer.*."
/>
>>                 </virtualDestinations>
>>             </virtualDestinationInterceptor>
>>         </destinationInterceptors>
>>
>>         <destinationPolicy>
>>             <policyMap>
>>                 <policyEntries>
>>                     <policyEntry topic=">"
>>                                  memoryLimit="16 mb"
>>                                  producerFlowControl="true">
>>                     </policyEntry>
>>                     <policyEntry queue=">"
>>                                  memoryLimit="16 mb"
>>                                  optimizedDispatch="true"
>>                                  producerFlowControl="true">
>>                     </policyEntry>
>>                 </policyEntries>
>>             </policyMap>
>>         </destinationPolicy>
>>
>>         <managementContext>
>>             <managementContext connectorPort="1101"
>>                                rmiServerPort="1100"
>>                                jmxDomainName="org.apache.activemq" />
>>         </managementContext>
>>
>>         <persistenceAdapter>
>>             <kahaDB directory="/data/activemq/"
>>                     enableIndexWriteAsync="true"
>>                     enableJournalDiskSyncs="false"
>>                     journalMaxFileLength="256mb" />
>>         </persistenceAdapter>
>>
>>         <systemUsage>
>>             <systemUsage>
>>                 <memoryUsage>
>>                     <memoryUsage limit="512 mb"/>
>>                 </memoryUsage>
>>                 <storeUsage>
>>                     <storeUsage limit="200 gb"/>
>>                 </storeUsage>
>>                 <tempUsage>
>>                     <tempUsage limit="1 gb"/>
>>                 </tempUsage>
>>             </systemUsage>
>>         </systemUsage>
>>
>>         <transportConnectors>
>>             <transportConnector name="nio"
>>
>> uri="nio://0.0.0.0:1102?transport.closeAsync=false" />
>>         </transportConnectors>
>>
>>     </broker>
>>
>>
>> Any suggestions would be appreciated.
>>
>> --
>> Best regards, Dmitriy V.
>
>
>
> --
> Best regards, Dmitriy V.



-- 
Best regards, Dmitriy V.

Mime
View raw message