activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rsteppac <ralf.steppac...@derivativepartners.com>
Subject ActiveMQ stalls once "Temp percent used" >= 100
Date Mon, 22 Jul 2013 14:12:49 GMT
Hello all,

I am having trouble getting the resource configuration right... My local
development server (ActiveMQ 5.8.0) stops to accept new messages
indefinitely once the "temp percent used" counter reaches >= 100%. From this
point onwards the server does not accept any new messages. Consumers are
served messages until "memory percent used" is down to 0%. The messages in
the temp space seemingly are not retrievable. Neither producers nor
consumers complain about timeouts or the server not being available at all.

Looking at a thread-dump of the server shows most NIO worker threads in this
state:

   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.activemq.usage.Usage.waitForSpace(Usage.java:97)
	- locked <0x00000000c0a0d520> (a java.lang.Object)
	at org.apache.activemq.usage.Usage.waitForSpace(Usage.java:85)
	at org.apache.activemq.usage.Usage.waitForSpace(Usage.java:75)
	at
org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.tryAddMessageLast(FilePendingMessageCursor.java:232)
	- locked <0x00000000c05941a8> (a
org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
	at
org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.addMessageLast(FilePendingMessageCursor.java:204)
	- locked <0x00000000c05941a8> (a
org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
	at
org.apache.activemq.broker.region.cursors.StoreQueueCursor.addMessageLast(StoreQueueCursor.java:96)
	- locked <0x00000000c05cfcf8> (a
org.apache.activemq.broker.region.cursors.StoreQueueCursor)
	at org.apache.activemq.broker.region.Queue.sendMessage(Queue.java:1683)
	at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:789)
	at org.apache.activemq.broker.region.Queue.send(Queue.java:721)
	at
org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:406)
	at
org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:392)
	at
org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:282)
	at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
	at
org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
	at
org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:317)
	at
org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135)
	at
org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:499)
	at
org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:749)
	at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:329)
	at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:184)
	at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
	at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
	at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:288)
	at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
	at
org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138)
	at
org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)
	at
org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)
	at
org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:679)

The resource configuration of the dev server is:

<systemUsage>
    <systemUsage>
        <memoryUsage>
            <memoryUsage limit="100 mb"/>
        </memoryUsage>
        <storeUsage>
            <storeUsage limit="200 mb"/>
        </storeUsage>
        <tempUsage>
            <tempUsage limit="200 mb"/>
        </tempUsage>
    </systemUsage>
</systemUsage>

Our production server behaves differently: Once the temp storage hits 100%
it starts to consume 100% CPU until it gets restarted. It needs to be
restarted twice as after the first restart it seems to read in a corrupted
temp storage state and immediately starts to consume 100% CPU again.

Please advise on how to avoid the server stall.

Some addtl. info: We only use non-persistent message. I cannot easily
increase the disk space for temp storage. I have producer flow control
switched off because unfortunately the messages are delivered in bursts.
Messages are set to expire after a couple of minutes which works OK for 99%
of the time. But once or twice a month we hit the 100% temp storage limit
and the server goes South.


Ralf



--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-stalls-once-Temp-percent-used-100-tp4669576.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message