activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Dulin <oleg.du...@gmail.com>
Subject Re: ActiveMQ stalls once "Temp percent used" >= 100
Date Mon, 22 Jul 2013 18:58:27 GMT
When messages are expired, they end up on the dead-letter queue.

If you don't have any consumers on that queue it fills up.

Take a look at this : 
http://activemq.apache.org/message-redelivery-and-dlq-handling.html

That may very well be what is filling up your storage.

Regards,
Oleg


On 2013-07-22 14:12:49 +0000, rsteppac said:

> Hello all,
> 
> I am having trouble getting the resource configuration right... My local
> development server (ActiveMQ 5.8.0) stops to accept new messages
> indefinitely once the "temp percent used" counter reaches >= 100%. From this
> point onwards the server does not accept any new messages. Consumers are
> served messages until "memory percent used" is down to 0%. The messages in
> the temp space seemingly are not retrievable. Neither producers nor
> consumers complain about timeouts or the server not being available at all.
> 
> Looking at a thread-dump of the server shows most NIO worker threads in this
> state:
> 
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at org.apache.activemq.usage.Usage.waitForSpace(Usage.java:97)
> 	- locked <0x00000000c0a0d520> (a java.lang.Object)
> 	at org.apache.activemq.usage.Usage.waitForSpace(Usage.java:85)
> 	at org.apache.activemq.usage.Usage.waitForSpace(Usage.java:75)
> 	at
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.tryAddMessageLast(FilePendingMessageCursor.java:232)

> 
> 	- locked <0x00000000c05941a8> (a
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
> 	at
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.addMessageLast(FilePendingMessageCursor.java:204)

> 
> 	- locked <0x00000000c05941a8> (a
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
> 	at
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.addMessageLast(StoreQueueCursor.java:96)

> 
> 	- locked <0x00000000c05cfcf8> (a
> org.apache.activemq.broker.region.cursors.StoreQueueCursor)
> 	at org.apache.activemq.broker.region.Queue.sendMessage(Queue.java:1683)
> 	at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:789)
> 	at org.apache.activemq.broker.region.Queue.send(Queue.java:721)
> 	at
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:406)
> 	at
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:392)
> 	at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:282)

> 
> 	at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
> 	at
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)

> 
> 	at
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:317)
> 	at
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135) 
> 
> 	at
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:499)

> 
> 	at
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:749)
> 	at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:329)

> 
> 	at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:184)

> 
> 	at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> 	at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)

> 
> 	at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:288)

> 
> 	at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83) 
> 
> 	at
> org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138) 
> 
> 	at
> org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)
> 	at
> org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)

> 
> 	at
> org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)
> 	at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> 	at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> 	at java.lang.Thread.run(Thread.java:679)
> 
> The resource configuration of the dev server is:
> 
> <systemUsage>
>     <systemUsage>
>         <memoryUsage>
>             <memoryUsage limit="100 mb"/>
>         </memoryUsage>
>         <storeUsage>
>             <storeUsage limit="200 mb"/>
>         </storeUsage>
>         <tempUsage>
>             <tempUsage limit="200 mb"/>
>         </tempUsage>
>     </systemUsage>
> </systemUsage>
> 
> Our production server behaves differently: Once the temp storage hits 100%
> it starts to consume 100% CPU until it gets restarted. It needs to be
> restarted twice as after the first restart it seems to read in a corrupted
> temp storage state and immediately starts to consume 100% CPU again.
> 
> Please advise on how to avoid the server stall.
> 
> Some addtl. info: We only use non-persistent message. I cannot easily
> increase the disk space for temp storage. I have producer flow control
> switched off because unfortunately the messages are delivered in bursts.
> Messages are set to expire after a couple of minutes which works OK for 99%
> of the time. But once or twice a month we hit the 100% temp storage limit
> and the server goes South.
> 
> 
> Ralf




-- 
Regards,
Oleg Dulin
http://www.olegdulin.com
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message