activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Posta <christian.po...@gmail.com>
Subject Re: Producer broker never resumes publishing after ResourceAllocationException
Date Tue, 12 Feb 2013 21:08:17 GMT
Well, looks like Broker B's store is only 512MB and broker A's store is up
to 15GB. If broker A is forwarding messages and broker B is storing them
(consumers cannot keep up), then your store limits will be reached on
broker B. This would explain the producer flow control stack trace you
posted from your logs.


On Tue, Feb 12, 2013 at 1:13 PM, vikramr <vikram.raman@gmail.com> wrote:

> Hi All. I'm not sure if a similar question has already been posted. I tried
> searching the forums and couldn't find any.
>
> We are using activemq 5.5.1 and have a producer on machine A publishing
> persistent messages to a local queue A (within broker A) that is bridged to
> a queue B (within broker B) on a machine B that a consumer reads from.
>
> Recently, the application got into a state where the consumer could not
> keep
> up with the producer and this led broker A to throw a
> javax.jms.ResourceAllocationException. It appears that the producer
> continued sending messages to the broker A, but just that the broker queued
> up these messages locally. But once we restarted broker B, essentially
> clearing its queues, broker A never seems to resume publishing the locally
> stored messages to broker B, instead continuing to queue them up under
> local
> storage. Restarting broker A however does resume the publishing if
> deleteAllMessagesOnStartup is set to false for the broker.
>
> I'd like to know if there is any configuration that could be changed to
> allow the broker A from resuming the publishing of messages once the
> consumer (broker B) was restarted. I have provided below the relevant
> snippets of the configuration and the log messages.
>
> *broker A configuration:*
>
> <beans>
> <broker xmlns="http://activemq.apache.org/schema/core"
>         brokerName="brokerA"
>         useJmx="true"
>         deleteAllMessagesOnStartup="true"
>         persistent="true"
>         advisorySupport="false"
>         >
>
>         <jmsBridgeConnectors>
>             <jmsQueueConnector
>
> outboundQueueConnectionFactory="#brokerBGeneralConnectionFactory">
>                 <outboundQueueBridges>
>                     <outboundQueueBridge
>                         outboundQueueName="Performance"
>                         localQueueName="Performance"/>
>                 </outboundQueueBridges>
>             </jmsQueueConnector>
>         </jmsBridgeConnectors>
>
>         <persistenceAdapter>
>             <amqPersistenceAdapter
> directory="${activemq.store.dir}/brokerA/persistent"
>             maxFileLength="32 MB"
>             cleanupInterval="2000"/>
>         </persistenceAdapter>
>
>         <systemUsage>
>             <systemUsage sendFailIfNoSpace="true">
>                 <memoryUsage>
>
>                     <memoryUsage limit="384 MB"/>
>                 </memoryUsage>
>                 <storeUsage>
>                     <storeUsage limit="15 GB"/>
>                 </storeUsage>
>                 <tempUsage>
>                     <tempUsage limit="384 MB"/>
>                 </tempUsage>
>             </systemUsage>
>         </systemUsage>
>
>
>
>         <transportConnectors>
>             <transportConnector name="brokerA.openwire.transport"
>                 uri="tcp://localhost:6116" />
>         </transportConnectors>
>     </broker>
>
>     <bean id="brokerAGeneralConnectionFactory"
>         class="org.apache.activemq.ActiveMQConnectionFactory">
>         <property name="brokerURL"
> value="failover://(tcp://brokerBHostName:6116)maxReconnectDelay-100" />
>     </bean>
> </beans>
>
> *broker B configuration:*
>
>     <broker xmlns="http://activemq.apache.org/schema/core"
>         brokerName="brokerB"
>         useJmx="true"
>         deleteAllMessagesOnStartup="true"
>         persistent="true"
>         advisorySupport="false"
>         >
>
>         <persistenceAdapter>
>             <amqPersistenceAdapter
> directory="${activemq.store.dir}/brokerB/persistent"
>             maxFileLength="32MB"
>             cleanupInterval="2000"/>
>         </persistenceAdapter>
>
>         <systemUsage>
>             <systemUsage sendFailIfNoSpace="true">
>                 <memoryUsage>
>                     <memoryUsage limit="512 MB"/>
>                 </memoryUsage>
>                 <storeUsage>
>                     <storeUsage limit="512 MB" />
>                 </storeUsage>
>                 <tempUsage>
>                     <tempUsage limit="512 MB"/>
>                 </tempUsage>
>             </systemUsage>
>         </systemUsage>
>
>
>         <transportConnectors>
>             <transportConnector name="brokerB.openwire.transport"
> uri="tcp://0.0.0.0:6116" />
>         </transportConnectors>
>
>     </broker>
>
> *Log of broker A*
> 2013-02-04 13:38:36,479 [ession Task-355] ERROR DestinationBridge - failed
> to forward message on attempt: 10 reason:
> javax.jms.ResourceAllocationException: Usage Manager Store is Full, 100% of
> 536870912. Stopping producer
> (ID:brokerAHostName-40967-1359993853867-1:13:1:1) to prevent flooding
> queue://Performance. See
> http://activemq.apache.org/producer-flow-control.html for more info
> message:
> ActiveMQObjectMessage {commandId = 294414, responseRequired = true,
> messageId = ID:brokerAHostName-40967-1359993853867-1:13:1:1:1,
> originalDestination = null, originalTransactionId = null, producerId =
> ID:brokerAHostName-37186-1359993892560-0:7:2:1, destination =
> queue://Performance, transactionId = null, expiration = 0, timestamp = 13
> 60003116477, arrival = 0, brokerInTime = 1360003107352, brokerOutTime =
> 1360003107352, correlationId = null, replyTo = null, persistent = true,
> type
> = null, priority = 4, groupID = null, groupSeq
> uence = 0, targetConsumerId = null, compressed = false, userID = null,
> content = org.apache.activemq.util.ByteSequence@3ac52ffa,
> marshalledProperties = org.apache.activemq.util.ByteSequence@67d0e
> 43a, dataStructure = null, redeliveryCounter = 0, size = 2627, properties =
> {PERF_FILENAME=PLUGIN_CSV.020420132030.log}, readOnlyProperties = true,
> readOnlyBody = true, droppable = false}
> javax.jms.ResourceAllocationException: Usage Manager Store is Full, 100% of
> 536870912. Stopping producer
> (ID:brokerAHostName-40967-1359993853867-1:13:1:1) to prevent flooding
> queue://Performance. See
> http://activemq.apache.org/producer-flow-control.html for more info
>         at
>
> org.apache.activemq.broker.region.BaseDestination.waitForSpace(BaseDestination.java:579)
>         at
> org.apache.activemq.broker.region.Queue.checkUsage(Queue.java:748)
>         at
> org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:674)
>         at org.apache.activemq.broker.region.Queue.send(Queue.java:653)
>         at
>
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:365)
>         at
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:523)
>         at
>
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
>         at
>
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:227)
>         at
>
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135)
>         at
>
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:458)
>         at
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:681)
>         at
>
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:306)
>         at
>
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
>         at
>
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
>         at
>
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
>         at
>
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:227)
>         at
>
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
>         at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:220)
>         at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202)
>         at java.lang.Thread.run(Thread.java:662)
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Producer-broker-never-resumes-publishing-after-ResourceAllocationException-tp4663276.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message