activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kevan Miller" <kevan.mil...@gmail.com>
Subject M4 Broker hang
Date Wed, 08 Mar 2006 19:26:10 GMT
Geronimo GBuild uses ActiveMQ to pass build requests/results between GBuild
agents. Twice, now, the M4 ActiveMQ server has become unresponsive and
required a restart. Unfortunately, logging information was not captured.
However, nohup output was captured and contained a thread dump.

I took a look and found two potential problems:

7 threads (the total number of expected clients) were blocked waiting for
memory as follows:

"tcp:///64.14.253.172:23076" daemon prio=1 tid=0xce5014a8 nid=0x3b83 in
Object.wait() [cd25f000..cd260238]
    at java.lang.Object.wait(Native Method)
    - waiting on <0xd229e7f8> (a java.lang.Object)
    at java.lang.Object.wait(Object.java:429)
    at org.apache.activemq.memory.UsageManager.waitForSpace(
UsageManager.java:83)
    - locked <0xd229e7f8> (a java.lang.Object)
    at org.apache.activemq.broker.region.Queue.send(Queue.java:229)
    at org.apache.activemq.broker.region.AbstractRegion.send(
AbstractRegion.java:178)
    at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java
:287)
    at org.apache.activemq.broker.TransactionBroker.send(
TransactionBroker.java:192)
    at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:94)
    at org.apache.activemq.broker.CompositeDestinationBroker.send(
CompositeDestinationBroker.java:97)
    at org.apache.activemq.broker.MutableBrokerFilter.send(
MutableBrokerFilter.java:108)
    at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:94)
    at org.apache.activemq.broker.AbstractConnection.processMessage(
AbstractConnection.java:332)
    at org.apache.activemq.command.ActiveMQMessage.visit(
ActiveMQMessage.java:508)
    at org.apache.activemq.broker.AbstractConnection.service(
AbstractConnection.java:185)
    at org.apache.activemq.broker.TransportConnection$1.onCommand(
TransportConnection.java:55)
    at org.apache.activemq.transport.ResponseCorrelator.onCommand(
ResponseCorrelator.java:82)
    at org.apache.activemq.transport.TransportFilter.onCommand(
TransportFilter.java:68)
    at org.apache.activemq.transport.WireFormatNegotiator.onCommand(
WireFormatNegotiator.java:105)
    at org.apache.activemq.transport.TransportSupport.doConsume(
TransportSupport.java:77)
    at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java
:136)
    at java.lang.Thread.run(Thread.java:534)

Also, two threads have a suspicious pattern of locks being held/waiting for:

"TcpTransportServer@tcp://stan.gbuild.org:16161" daemon prio=1
tid=0xced70460 nid=0x3b83 in Object.wait() [cdef8000..cdef9238]
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:429)
    at edu.emory.mathcs.backport.java.util.concurrent.CountDownLatch.await(
CountDownLatch.java:173)
    - locked <0xd269e8b0> (a
edu.emory.mathcs.backport.java.util.concurrent.CountDownLatch)
    at org.apache.activemq.transport.WireFormatNegotiator.oneway(
WireFormatNegotiator.java:61)
    at org.apache.activemq.transport.MutexTransport.oneway(
MutexTransport.java:44)
    - locked <0xd269e8f8> (a java.lang.Object)
    at org.apache.activemq.transport.ResponseCorrelator.oneway(
ResponseCorrelator.java:54)
    at org.apache.activemq.broker.TransportConnection.dispatch(
TransportConnection.java:189)
    at org.apache.activemq.broker.AbstractConnection.start(
AbstractConnection.java:133)
    at org.apache.activemq.broker.TransportConnection.start(
TransportConnection.java:70)
    at org.apache.activemq.broker.TransportConnector$1.onAccept(
TransportConnector.java:128)
    at org.apache.activemq.transport.tcp.TcpTransportServer.run(
TcpTransportServer.java:105)
    at java.lang.Thread.run(Thread.java:534)

and

"tcp:///64.14.253.172:21634" daemon prio=1 tid=0x0807cee0 nid=0x3b83 waiting
for monitor entry [cd0dc000..cd0dd238]
    at org.apache.activemq.transport.MutexTransport.oneway(
MutexTransport.java:43)
    - waiting to lock <0xd269e8f8> (a java.lang.Object)
    at org.apache.activemq.transport.ResponseCorrelator.oneway(
ResponseCorrelator.java:54)
    at org.apache.activemq.broker.TransportConnection.stop(
TransportConnection.java:76)
    at org.apache.activemq.broker.jmx.ManagedTransportConnection.stop(
ManagedTransportConnection.java:62)
    at org.apache.activemq.util.ServiceSupport.dispose(ServiceSupport.java
:33)
    at
org.apache.activemq.broker.AbstractConnection.serviceTransportException(
AbstractConnection.java:160)
    at org.apache.activemq.broker.TransportConnection$1.onException(
TransportConnection.java:61)
    at org.apache.activemq.transport.TransportFilter.onException(
TransportFilter.java:102)
    at org.apache.activemq.transport.TransportFilter.onException(
TransportFilter.java:102)
    at org.apache.activemq.transport.TransportFilter.onException(
TransportFilter.java:102)
    at org.apache.activemq.transport.TransportSupport.onException(
TransportSupport.java:90)
    at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java
:149)
    at java.lang.Thread.run(Thread.java:534)

--kevan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message