Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: neutral (athena.apache.org: 216.139.250.139 is neither permitted
 nor denied by domain of mrudolph@gmail.com)
Date: Tue, 24 Jul 2012 13:45:14 -0700 (PDT)
From: secondspan <mrudolph@gmail.com>
To: users@activemq.apache.org
Message-ID: <1343162714884-4654328.post@n4.nabble.com>
Subject: ActiveMQ 5.6 CPU steadily increases but drops at noon and midnight
 exactly?
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Hi all,

We've been having a strange issue where the CPU load on our ActiveMQ server
increases continuously, only to suddenly drop off like clockwork at noon and
midnight sharp.  We are using JDK 1.7_04 with the G1 collector.

The server is receiving barely any traffic from 8pm-5am, so the CPU load
appears completely unrelated to the rate at which we process messages
through ActiveMQ.   Also, 10am-4pm is when it is processing the most
messages, so it's odd that it's at the lowest CPU usage around 12:10pm in
the middle of this peak time.

At first I suspected run-away NIO threads that are furiously polling, but it
does appear that ActiveMQ is cleaning up and constantly re-allocating new
"NIO Worker" threads and in examining their stacktrace they are all happily
spending most of their time here:

sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)


Here you can see the "Top Threads" using CPU as reported by the top threads
plug-in for JConsole.  The top threads are always mostly the "NIO Worker"
threads.

http://activemq.2283324.n4.nabble.com/file/n4654328/alertmq01_nio_cpu_12hrs.jpg 

Here you can see like clockwork the CPU drops at noon and midnight.  The
23rd is the exception, where the drop-offs that didn't happen at noon and
midnight were manual restarts of the server trying various additional
settings.

http://activemq.2283324.n4.nabble.com/file/n4654328/cpu-week.png 


For kicks, I tried the 5.7-SNAPSHOT from July 23rd.  It didn't affect the
gradual increase in CPU load.

We are using "-Dorg.apache.activemq.UseDedicatedTaskRunner=false"

Here's the line from activemq.xml where we define the transportConnector:

<transportConnector name="openwire"
uri="nio://0.0.0.0:61616?transport.closeAsync=false&amp;soTimeout=60000&amp;wireFormat.maxInactivityDurationInitalDelay=60000&amp;keepAlive=true"/>


I'm likely to try reverting back to BIO next as I have a hunch this might be
similar to the NIO problems that Jetty used to have back on early JDK 1.6
flavors.

I figure to try the forums to see if anyone had any thoughts...

Cheers,
Mike


--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-5-6-CPU-steadily-increases-but-drops-at-noon-and-midnight-exactly-tp4654328.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.