activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ItBurn <itbu...@hotmail.com>
Subject Thread spike/leak when broker JVM fails to compete with consumer JVM
Date Wed, 25 Feb 2009 22:23:17 GMT

We have an application that has a broker(ActiveMQ 5.2, durable, in an Oracle
DB) and 14 consumers (On various queues/topics) in a single JVM .  We also
have an outside module that runs in it's own JVM and houses a consumer on
another queue.  The outside module needs the main application's broker.

Our application can also be deployed as a cluster with each machine having
copies of both the main application and the module.  The extra brokers
become slaves to the master broker (first application that was started).

The cluster we test this setup on has several single core(important because
on dual cores both JVMs are not competing) Windows XP machines with 2gigs of
ram and java 1.6.

The problem is that when the consumer of the outside module on the machine
with the master broker receives a message, it starts doing heavy processing
that takes 99% of the cpu and the broker seems to stop processing while this
is going on.  After the module is done, the broker wakes up and creates a
large spike of threads (seen with the jconsole).  With a cluster of 8
machines, it can easily create 400 threads and give us
"java.lang.OutOfMemoryError: unable to create new native thread".  These
threads are simple Transport threads:

Name: ActiveMQ Transport: tcp:///10.0.0.5:61616
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
java.io.DataInputStream.readInt(DataInputStream.java:370)
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:209)
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:201)
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:184)
java.lang.Thread.run(Thread.java:619)

If we don't get the memory error, shortly after the spike(1-2 seconds), most
threads disappear, but the new total is higher than before suggesting that
there might have been a thread leak.  Running the test again several times
will eventually bust the memory.

I've tried enabling thread pooling in ActiveMQ with no effect.  The problem
does not happen if I move the module consumer inside the main
application(probably because the JVM is not competing with another).  I've
run out of ideas on how to fix this.  Playing with the XMX and XMS settings
isn't an option.  Any idea on what could help or why this is happening? 
Maybe some configuration I didn't find?

I reproduced the problem using the ActiveMQ examples.  It's easy to see the
thread spike with only two machines in a cluster.  I'll attach the modified
example in this message.

Extract the zip into the example folder.  Read the Readme.txt file for
details on how to run the test.
http://www.nabble.com/file/p22213287/example.zip example.zip 
-- 
View this message in context: http://www.nabble.com/Thread-spike-leak-when-broker-JVM-fails-to-compete-with-consumer-JVM-tp22213287p22213287.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message