activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: How to avoid blocking of queue browsing after ActiveMQ checkpoint call
Date Fri, 08 Jan 2016 15:32:57 GMT
If you increase your JVM size (4GB, 8GB, etc., the biggest your OS and
hardware will support), does the behavior change?  Does it truly take all
available memory, or just all the memory that you've made available to it
(which isn't tiny but really isn't all that big)?

Also, how do you know that the
MessageCursor seems to decide that there is not enough memory and stops
delivery of queue content to browsers/consumers?  What symptom tells you
that?
On Jan 8, 2016 8:25 AM, "Klaus Pittig" <klaus.pittig@futura4retail.com>
wrote:

> (related issue: https://issues.apache.org/jira/browse/AMQ-6115)
>
> There's a problem when Using ActiveMQ with a large number of Persistence
> Queues (250) á 1000 persistent TextMessages á 10 KB.
>
> Our scenario requires these messages to remain in the storage over a
> long time (days), until they are consumed (large amounts of data are
> staged for distribution for many consumer, that could be offline for
> some days).
>
>
> After the Persistence Store is filled with these Messages and after a
> broker restart we can browse/consume some Queues  _until_ the
> #checkpoint call after 30 seconds.
>
> This call causes the broker to use all available memory and never
> releases it for other tasks such as Queue browse/consume. Internally the
> MessageCursor seems to decide, that there is not enough memory and stops
> delivery of queue content to browsers/consumers.
>
>
> => Is there a way to avoid this behaviour by configuration or is this a
> bug?
>
> The expectation is, that we can consume/browse any queue under all
> circumstances.
>
> Settings below are in production for some time now and several
> recommendations are applied found in the ActiveMQ documentation
> (destination policies, systemUsage, persistence store options etc.)
>
>  - Behaviour is tested with ActiveMQ: 5.11.2, 5.13.0 and 5.5.1.
>  - Memory Settings: Xmx=1024m
>  - Java: 1.8 or 1.7
>  - OS: Windows, MacOS, Linux
>  - PersistenceAdapter: KahaDB or LevelDB
>  - Disc: enough free space (200 GB) and physical memory (16 GB max).
>
> Besides the above mentioned settings we use the following settings for
> the broker (btw: changing the memoryLimit to a lower value like 1mb does
> not change the situation):
>
>     <destinationPolicy>
>         <policyMap>
>             <policyEntries>
>                 <policyEntry queue=">" producerFlowControl="false"
> optimizedDispatch="true" memoryLimit="128mb"
> timeBeforeDispatchStarts="1000">
>                     <dispatchPolicy>
>                         <strictOrderDispatchPolicy />
>                     </dispatchPolicy>
>                     <pendingQueuePolicy>
>                         <storeCursor />
>                     </pendingQueuePolicy>
>                 </policyEntry>
>             </policyEntries>
>         </policyMap>
>     </destinationPolicy>
>     <systemUsage>
>         <systemUsage sendFailIfNoSpace="true">
>             <memoryUsage>
>                 <memoryUsage limit="50 mb" />
>             </memoryUsage>
>             <storeUsage>
>                 <storeUsage limit="80000 mb" />
>             </storeUsage>
>             <tempUsage>
>                 <tempUsage limit="1000 mb" />
>             </tempUsage>
>         </systemUsage>
>     </systemUsage>
>
> If we set the **cursorMemoryHighWaterMark** in the destinationPolicy to
> a higher value like **150** or **600** depending on the difference
> between memoryUsage and the available heap space relieves the situation
> a bit for a workaround, but this is not really an option for production
> systems in my point of view.
>
> Screenie with information from Oracle Mission Control showing those
> ActiveMQTextMessage instances that are never released from memory:
>
> http://goo.gl/EjEixV
>
>
> Cheers
> Klaus
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message