activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Illtud Daniel <illtud.dan...@llgc.org.uk>
Subject Large messages, slow consumers and java heap space
Date Mon, 07 Mar 2011 18:19:26 GMT
I'm using stomp, java clients and camel to move messages through
a 'pipeline' of activemq queues. The messages can be very big
(>120MB) and unfortunately the OS is 32-bit, so I want to keep
memory usage to a minimum (2GB limit). There isn't a huge number
of messages, but I keep running out of heap space. Each queue has
only one producer and one consumer.

I've attempted to read everything I can about producer flow control,
message cursors, prefetch limits, etc. and I still can't get it
to do what I want. I don't care about speed at all, I have plenty
of disk space, and all messages are persistent.

The JVM has the max 2024MB of memory.

bits of activemq.xml:

...
                 <policyEntry queue=">" producerFlowControl="true" 
memoryLimit="150mb" queuePrefetch="1">
                   <pendingQueuePolicy>
                     <fileQueueCursor />
                   </pendingQueuePolicy>

                 </policyEntry>
...

         <systemUsage>
             <systemUsage>
                 <memoryUsage>
                     <memoryUsage limit="1024 mb"/>
                 </memoryUsage>
                 <storeUsage>
                     <storeUsage limit="25 gb"/>
                 </storeUsage>
                 <tempUsage>
                     <tempUsage limit="5 gb"/>
                 </tempUsage>
             </systemUsage>
         </systemUsage>


I've tried it with and without producerFlowControl. I get stuff like:

2011-03-07 18:11:28,004 | ERROR | Failed to page in more queue messages 
  | org.apache.activemq.broker.region.Queue | Queue:scif_mets_pre_jhove_in
java.lang.OutOfMemoryError: Java heap space
	at 
org.apache.activemq.openwire.v6.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:639)

and

2011-03-07 18:17:25,011 | WARN  | Execution of JMS message listener 
failed, and no ErrorHandler has been set. | 
org.springframework.jms.listener.DefaultMessageListenerContainer | 
DefaultMessageListenerContainer-1
java.lang.OutOfMemoryError: Java heap space


Anybody help me? How can I keep the memory usage to a minimum, at
the expense of performance, disk space, anything else?

Am I just expecting too much because my messages are too big?

-- 
Illtud Daniel                                 illtud.daniel@llgc.org.uk
Prif Swyddog Technegol                          Chief Technical Officer
Llyfrgell Genedlaethol Cymru                  National Library of Wales

Mime
View raw message