Have you checked out 
http://activemq.apache.org/message-cursors.html

4.1.1 seems to store some kind of message reference in memory so broker may
run out of memory even if persistence is enabled.


Maarten Dirkse-2 wrote:
> 
> Allright, the sleuthing continues. I did a heap dump of the broker vm
> when the broker was using 100% of its memory and everything had come to
> a standstill. I openend the heapdum in visualvm (see pic), but I have to
> confess it doesn't mean that much to me. There seem to be an awful lot
> of ActiveMQTextMesssages hanging around, and ObjectMessages as well. Not
> sure what they're all doing in memory, as I'd rather see then on disk
> (in the case of queues) or discarded (in the case of topics).
> Anyone have any ideas why my broker is maxing out?
> Regards,
> Maarten 
> 
> -----Original Message-----
> From: Maarten Dirkse [mailto:Maarten.Dirkse@filterworks.com] 
> Sent: 2 mei 2008 16:45
> To: users@activemq.apache.org
> Subject: RE: Broker hitting max memory with large persistent queue
> 
> Hi Joe,
> 
> Using the Spring listeners, we've actually got about 10-20 threads on
> each stage of the pipeline, all consuming messages. The problem is that
> we'd like to be able to deal with a situation where there are
> potentially millions of messages waiting in to be processed without
> having to concede ever more ram to the broker (or ever more threads to
> the consumers). New messages should just get in line to be processed,
> whether that's 100 milliseconds or 100 minutes from the time they
> arrive. I thought persistence would do this for us, as it writes the
> messages to disk instead of keeping them in ram, but the broker is still
> running out of ram.
> 
> It seems as though, for some reason, it's keeping something in memory
> for every message in a persistent queue, and once it hits 100% memory
> usage it's game over.
> 
> Regards,
> Maarten
> 
> -----Original Message-----
> From: ttmdev [mailto:joe.fernandez@ttmsolutions.com]
> Sent: 2 mei 2008 16:22
> To: users@activemq.apache.org
> Subject: RE: Broker hitting max memory with large persistent queue
> 
> 
> What about creating clusters of consumers for each of your processing
> stages to better-handle the load spikes? The messages will get
> load-balanced across the consumers in a cluster.
> 
> Joe
> Get a free ActiveMQ user guide at www.ttmsolutions.com  
> 
> 
> 
> Maarten Dirkse-2 wrote:
>> 
>> An update:
>> I've now tried it with the standard amqpersistence config:
>> 
>>     <persistenceAdapter>
>>       <amqPersistenceAdapter directory="/var/filterworks/amq/store"
>> maxFileLength="32mb"/>
>>     </persistenceAdapter>
>> 
>> And it still results in the same problem. When I give the broker 128mb
> 
>> of memory, everything runs fine until it has processed about 126,000 
>> messages. Then the memory usage hits 100%, and everything slows to a 
>> crawl.
>> 
>> Anyone have any ideas as to what's going on?
>> 
>> Regards,
>> Maarten
>> 
>> -----Original Message-----
>> From: Maarten Dirkse [mailto:Maarten.Dirkse@filterworks.com]
>> Sent: 2 mei 2008 9:02
>> To: users@activemq.apache.org
>> Subject: Broker hitting max memory with large persistent queue
>> 
>> Hi,
>> I've spent quite a bit of time trying to find a solution to this 
>> problem, but nothing that I've come across in the documentation or the
> 
>> mailing list seems to help, so here goes:
>> 
>> I've got ActiveMQ set up to provide messaging between a chain of four 
>> components, connected by three queues like so:
>> Component 1 --> queue A --> component 2 --> queue B --> component 3 
>> --> queue C --> component 4
>> 
>> Component 1 gets things going by messages and putting them on queue A.
>> Component 2 then processes these, and, for every message received, 
>> puts a message on queue B. Component three then does the same thing as
> 
>> 2, generating an equal amount of message for component 4. Beside the 
>> queue pipeline, components also communicate with each other using a
> "command"
>> topic which produces 2 messages for every queue message that goes 
>> through the pipeline.
>> All queues are (pure JDBC) persistent, topics are non-persistent.
>> 
>> This setup works very well for low volumes of messages. At a certain 
>> point though, the first component has to start producing messages at 
>> rates of several thousand per minute. Component 2 can't consume the 
>> messages at the rate they're produced, so the size of queue A slowly 
>> grows into the thousands.
>> 
>> The problem is that, once the size of queue A gets large enough, the 
>> broker runs out of memory and everything will screech to a halt (ie 
>> messages will be processed at a rate of 1 per minute or something). At
> 
>> around 256mb of broker memory for example, it reaches 100% usage when 
>> queue A contains around 16.000 messages. The number of messages that 
>> queue A can contain before activemq becomes unusably slow grows and 
>> shrinks according to the amount of memory the broker has.
>> 
>> What I don't get is why the broker slowly uses up all its memory for a
> 
>> queue that persists its messages to the database (this actually 
>> happens, the database does fill up). I've read a number of times in 
>> the documentation and mailing list that AMQ is capable of dealing with
> 
>> queues that contain millions of messages, but in my setup it grinds to
> 
>> a halt at a number far below that.
>> 
>> Can anyone tell me what I'm doing wrong?
>> 
>> Below is the setup used for the broker. Queues are defined in Spring 
>> using <amq:queue> and producers and consumers use the Spring JMS 
>> template and JMS Listeners respectively. I'm using AMQ 5.1 with 
>> Postgresql for persistence and Spring 2.5.3 to configure everything.
>> 
>> Broker config:
>> <broker id="broker" useJmx="true" brokerName="${brokerA.name}"
>> xmlns="http://activemq.apache.org/schema/core"
>> dataDirectory="${activemq.data}">
>> 
>>   <persistenceAdapter>
>>     <jdbcPersistenceAdapter dataDirectory="${activemq.data}/jdbc"
>> dataSource="#postgres-ds" createTablesOnStartup="true" />
>>   </persistenceAdapter>
>> 
>>   <systemUsage>
>>     <systemUsage>
>>       <memoryUsage>
>>         <memoryUsage limit="128mb" />
>>       </memoryUsage>
>>     </systemUsage>
>>   </systemUsage>
>> 
>>   <transportConnectors>
>>     <transportConnector uri="tcp://localhost:61616" />
>>   </transportConnectors>
>> </broker>
>> 
>> Template and queue/topic config:
>> 
>> <beans>
>>   <amq:connectionFactory brokerURL="${broker.jmsUrl}" id="jmsFactory"
>> dispatchAsync="true" />
>> 
>>   <bean id="scf"
>> class="org.springframework.jms.connection.SingleConnectionFactory"
>> p:targetConnectionFactory-ref="jmsFactory" />
>> 
>>   <bean class="org.springframework.jms.core.JmsTemplate"
>> id="jmsTemplate" p:receiveTimeout="${jms.receivetimeout}"
>> p:connectionFactory-ref="scf" />
>> 
>>   <bean class="org.springframework.jms.core.JmsTemplate"
>> id="commandTemplate" p:receiveTimeout="${jms.receivetimeout}"
>> p:explicitQosEnabled="true"
>>     p:priority="4" p:deliveryMode="2" p:timeToLive="120000"
>> p:connectionFactory-ref="scf" p:defaultDestination-ref="commandTopic" 
>> />
>> 
>>   <amq:queue id="A" physicalName="queue.A" />
>>   <amq:queue id="B" physicalName="queue.B" />
>>   <amq:queue id="C" physicalName="queue.C" />
>> 
>>   <amq:topic id="commandTopic" physicalName="command.topic" /> 
>> </beans>
>> 
>> Thanks in advance,
>> Maarten
>> 
>> 
>> 
>> 
> 
> --
> View this message in context:
> http://www.nabble.com/Broker-hitting-max-memory-with-large-persistent-qu
> eue-tp17014805s2354p17020505.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Broker-hitting-max-memory-with-large-persistent-queue-tp17014805s2354p17052388.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.