activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tpounds <trevor.pou...@gmail.com>
Subject Re: Broker hitting max memory with large persistent queue
Date Mon, 02 Jun 2008 17:30:27 GMT



Maarten Dirkse-2 wrote:
> 
> Hi,
> I've spent quite a bit of time trying to find a solution to this problem,
> but nothing that I've come across in the documentation or the mailing list
> seems to help, so here goes:
> 
> I've got ActiveMQ set up to provide messaging between a chain of four
> components, connected by three queues like so:
> Component 1 --> queue A --> component 2 --> queue B --> component 3 -->
> queue C --> component 4
> 
> Component 1 gets things going by messages and putting them on queue A.
> Component 2 then processes these, and, for every message received, puts a
> message on queue B. Component three then does the same thing as 2,
> generating an equal amount of message for component 4. Beside the queue
> pipeline, components also communicate with each other using a "command"
> topic which produces 2 messages for every queue message that goes through
> the pipeline.
> All queues are (pure JDBC) persistent, topics are non-persistent.
> 
> This setup works very well for low volumes of messages. At a certain point
> though, the first component has to start producing messages at rates of
> several thousand per minute. Component 2 can't consume the messages at the
> rate they're produced, so the size of queue A slowly grows into the
> thousands.
> 
> The problem is that, once the size of queue A gets large enough, the
> broker runs out of memory and everything will screech to a halt (ie
> messages will be processed at a rate of 1 per minute or something). At
> around 256mb of broker memory for example, it reaches 100% usage when
> queue A contains around 16.000 messages. The number of messages that queue
> A can contain before activemq becomes unusably slow grows and shrinks
> according to the amount of memory the broker has.
> 
> What I don't get is why the broker slowly uses up all its memory for a
> queue that persists its messages to the database (this actually happens,
> the database does fill up). I've read a number of times in the
> documentation and mailing list that AMQ is capable of dealing with queues
> that contain millions of messages, but in my setup it grinds to a halt at
> a number far below that.
> 
> Can anyone tell me what I'm doing wrong?
> 
> Below is the setup used for the broker. Queues are defined in Spring using
> <amq:queue> and producers and consumers use the Spring JMS template and
> JMS Listeners respectively. I'm using AMQ 5.1 with Postgresql for
> persistence and Spring 2.5.3 to configure everything.
> 
> Broker config:
> <broker id="broker" useJmx="true" brokerName="${brokerA.name}"
> xmlns="http://activemq.apache.org/schema/core"
> dataDirectory="${activemq.data}">
> 
>   <persistenceAdapter>
>     <jdbcPersistenceAdapter dataDirectory="${activemq.data}/jdbc"
> dataSource="#postgres-ds" createTablesOnStartup="true" />
>   </persistenceAdapter>
> 
>   <systemUsage>
>     <systemUsage>
>       <memoryUsage>
>         <memoryUsage limit="128mb" />
>       </memoryUsage>
>     </systemUsage>
>   </systemUsage>
> 
>   <transportConnectors>
>     <transportConnector uri="tcp://localhost:61616" />
>   </transportConnectors>
> </broker>
> 
> Template and queue/topic config:
> 
> <beans>
>   <amq:connectionFactory brokerURL="${broker.jmsUrl}" id="jmsFactory"
> dispatchAsync="true" />
> 
>   <bean id="scf"
> class="org.springframework.jms.connection.SingleConnectionFactory"
> p:targetConnectionFactory-ref="jmsFactory" />
> 
>   <bean class="org.springframework.jms.core.JmsTemplate" id="jmsTemplate"
> p:receiveTimeout="${jms.receivetimeout}" p:connectionFactory-ref="scf" />
> 
>   <bean class="org.springframework.jms.core.JmsTemplate"
> id="commandTemplate" p:receiveTimeout="${jms.receivetimeout}"
> p:explicitQosEnabled="true"
>     p:priority="4" p:deliveryMode="2" p:timeToLive="120000"
> p:connectionFactory-ref="scf" p:defaultDestination-ref="commandTopic" />
> 
>   <amq:queue id="A" physicalName="queue.A" />
>   <amq:queue id="B" physicalName="queue.B" />
>   <amq:queue id="C" physicalName="queue.C" />
> 
>   <amq:topic id="commandTopic" physicalName="command.topic" />
> </beans>
> 
> Thanks in advance,
> Maarten
> 
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Broker-hitting-max-memory-with-large-persistent-queue-tp17014805s2354p17606223.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message