activemq-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From build...@apache.org
Subject svn commit: r943334 - in /websites/production/activemq/content: cache/main.pageCache how-do-i-configure-activemq-to-hold-100s-of-millions-of-queue-messages-.html scaling-the-depth-of-a-queue.html
Date Wed, 11 Mar 2015 10:39:02 GMT
Author: buildbot
Date: Wed Mar 11 10:39:02 2015
New Revision: 943334

Log:
Production update by buildbot for activemq

Modified:
    websites/production/activemq/content/cache/main.pageCache
    websites/production/activemq/content/how-do-i-configure-activemq-to-hold-100s-of-millions-of-queue-messages-.html
    websites/production/activemq/content/scaling-the-depth-of-a-queue.html

Modified: websites/production/activemq/content/cache/main.pageCache
==============================================================================
Binary files - no diff available.

Modified: websites/production/activemq/content/how-do-i-configure-activemq-to-hold-100s-of-millions-of-queue-messages-.html
==============================================================================
--- websites/production/activemq/content/how-do-i-configure-activemq-to-hold-100s-of-millions-of-queue-messages-.html
(original)
+++ websites/production/activemq/content/how-do-i-configure-activemq-to-hold-100s-of-millions-of-queue-messages-.html
Wed Mar 11 10:39:02 2015
@@ -32,16 +32,6 @@
     </style>
     <![endif]-->
 
-          <link href='http://activemq.apache.org/styles/highlighter/styles/shCore.css'
rel='stylesheet' type='text/css' /> 
-      <link href='http://activemq.apache.org/styles/highlighter/styles/shThemeEclipse.css'
rel='stylesheet' type='text/css' /> 
-      <script src='http://activemq.apache.org/styles/highlighter/scripts/shCore.js' type='text/javascript'></script>

-              <script src='http://activemq.apache.org/styles/highlighter/scripts/shBrushJava.js'
type='text/javascript'></script> 
-              <script src='http://activemq.apache.org/styles/highlighter/scripts/shBrushPlain.js'
type='text/javascript'></script> 
-         
-      <script type="text/javascript"> 
-        SyntaxHighlighter.defaults['toolbar'] = false; 
-        SyntaxHighlighter.all(); 
-      </script> 
     
     <title>
     Apache ActiveMQ &#8482; -- How do I configure ActiveMQ to hold 100s of millions of
Queue Messages ?
@@ -82,97 +72,7 @@
   <tbody>
         <tr>
         <td valign="top" width="100%">
-<div class="wiki-content maincontent">Nearly all messaging systems (certainly open
source ones) hold either a copy of a persistent message or a reference to a persisted message
in memory. This is primarily to try and improve performance, but it also can significantly
decrease the complexity of implementation. In fact ActiveMQ version 4 and below worked this
- way - by holding references to persisted messages in memory.<p>However there is a
limitation to this approach, no matter how much memory you have at your disposal, you will
hit a limit to the number persistent messages a broker can handle at any particular time.</p><p>To
get around this limitation, ActiveMQ introduced a paging cache - for all message stores (except
the memory store) to get the best of both worlds - great performance and the ability to hold
100s of millions of messages in persistent store. ActiveMQ is <strong>not</strong>
limited by memory availability, but by the size of the disk available to hold the persisten
 t messages.</p><p>For more information - see below:</p><p></p><h2
id="HowdoIconfigureActiveMQtohold100sofmillionsofQueueMessages?-MessageCursors">Message
Cursors</h2>
-
-<p>A common problem in previous versions of ActiveMQ was <a shape="rect" href="my-producer-blocks.html">running
out of RAM buffer</a> when using non-persistent messaging.</p>
-
-<p>Beginning with ActiveMQ 5.0.0, there is a new memory model that allows messages
to be paged in from storage when space is available (using Store cursors for persistent messages).</p>
-
-<p>Releases prior to 5.0 kept references in memory for all the messages that could
be dispatched to an active Durable Topic Consumer or a Queue. While a reference itself is
not large, it does impose a limit on the maximum number of messages that can be pending delivery.</p>
-
-<p>A typical approach for messaging systems dispatching persistent messages is to pull
them in batches from long term storage when a client is ready to consume them, using a cursor
to maintain the next to dispatch position. This is a robust and very scalable approach, but
not the most performant for cases when the consumer(s) can keep up with the producer(s) of
messages.</p>
-
-<p>ActiveMQ 5.0 takes a hybrid approach, allowing messages to pass from producer to
consumer directly (after the messages have been persisted), but switches back to using cursors
if the consumer(s) fall behind.</p>
-
-<p>When Message Consumers are both active and fast - keeping up with the Message Producer(s)
- messages are stored and then passed to a dispatch queue in the broker associated with the
Consumer: <img class="confluence-embedded-image image-center" src="how-do-i-configure-activemq-to-hold-100s-of-millions-of-queue-messages-.data/DispatchFastConsumers.png"
data-image-src="/confluence/download/attachments/48811/DispatchFastConsumers.png?version=1&amp;modificationDate=1193437635000&amp;api=v2"><br
clear="none">
-If a Consumer becomes active after messages are pending from the store for it, or it's slower
than the producer, then messages are paged in to the dispatch queue from a pending cursor:
<img class="confluence-embedded-image image-center" src="how-do-i-configure-activemq-to-hold-100s-of-millions-of-queue-messages-.data/DispatchSlowConsumers.png"
data-image-src="/confluence/download/attachments/48811/DispatchSlowConsumers.png?version=1&amp;modificationDate=1193437656000&amp;api=v2"></p>
-
-<h3 id="HowdoIconfigureActiveMQtohold100sofmillionsofQueueMessages?-TypesofCursor">Types
of Cursor</h3>
-
-<p>The default message cursor type in ActiveMQ 5.0&#160;is Store based. &#160;It
behaves as above. There are two&#160;additional&#160;types of cursor that could be
used: <strong>VM Cursor</strong> and <strong>File based Cursor</strong>,
described below.</p>
-
-<h4 id="HowdoIconfigureActiveMQtohold100sofmillionsofQueueMessages?-VMCursor">VM Cursor</h4>
-
-<p>The VM Cursor is how ActiveMQ 4.x works: references to a message are held in memory,
and passed to the dispatch queue when needed. This can be very fast, but also has the downside
of not being able to handle very slow consumers or consumers that have been inactive for a
long time: <img class="confluence-embedded-image image-center" src="how-do-i-configure-activemq-to-hold-100s-of-millions-of-queue-messages-.data/VMCursor.png"
data-image-src="/confluence/download/attachments/48811/VMCursor.png?version=1&amp;modificationDate=1193439154000&amp;api=v2"></p>
-
-<h4 id="HowdoIconfigureActiveMQtohold100sofmillionsofQueueMessages?-FilebasedCursor">File
based Cursor</h4>
-
-<p>The File based Cursor is dervied from the VM Cursor. &#160;When memory in the
broker reaches its limit, it can page messages to temporary files on disk. This type of cursor
can be used when the message store might be relatively slow, but consumers are generally fast.
By buffering to disk, it allows the message broker to handle message bursts from producers
without resorting to paging in from slow storage: <img class="confluence-embedded-image
image-center" src="how-do-i-configure-activemq-to-hold-100s-of-millions-of-queue-messages-.data/FileCursor.png"
data-image-src="/confluence/download/attachments/48811/FileCursor.png?version=1&amp;modificationDate=1193438877000&amp;api=v2"></p>
-
-<h4 id="HowdoIconfigureActiveMQtohold100sofmillionsofQueueMessages?-PagingforNon-PersistentMessages">Paging
for Non-Persistent Messages</h4>
-
-<p>The store based cursor also handles cursors for non-persistent messages, which are
not stored in the message store. Non-persistent messages are passed directly to the cursor,
so the store based cursor embeds a file based cursor just for these types of messages: <img
class="confluence-embedded-image image-center" src="how-do-i-configure-activemq-to-hold-100s-of-millions-of-queue-messages-.data/NonPersistentMsgs.png"
data-image-src="/confluence/download/attachments/48811/NonPersistentMsgs.png?version=1&amp;modificationDate=1193445319000&amp;api=v2"></p>
-
-<h3 id="HowdoIconfigureActiveMQtohold100sofmillionsofQueueMessages?-ConfiguringCursors">Configuring
Cursors</h3>
-
-<p>By default, Store based cursors are used, but it is possible to configure different
cursors depending on the destination.</p>
-
-<h4 id="HowdoIconfigureActiveMQtohold100sofmillionsofQueueMessages?-Topicsubscribers">Topic
subscribers</h4>
-
-<p>For Topics there is a dispatch queue and pending cursor for every subscriber. &#160;It's
possible to configure different policies for durable subscribers and transient subscribers
- e.g:</p>
-<div class="code panel pdl" style="border-width: 1px;"><div class="codeContent panelContent
pdl">
-<script class="theme: Default; brush: java; gutter: false" type="syntaxhighlighter"><![CDATA[
-&lt;destinationPolicy&gt;
-      &lt;policyMap&gt;
-        &lt;policyEntries&gt;
-          &lt;policyEntry topic=&quot;org.apache.&gt;&quot; producerFlowControl=&quot;false&quot;
memoryLimit=&quot;1mb&quot;&gt;
-            &lt;dispatchPolicy&gt;
-              &lt;strictOrderDispatchPolicy /&gt;
-            &lt;/dispatchPolicy&gt;
-            &lt;deadLetterStrategy&gt;
-              &lt;individualDeadLetterStrategy  topicPrefix=&quot;Test.DLQ.&quot;
/&gt;
-            &lt;/deadLetterStrategy&gt;
-            &lt;pendingSubscriberPolicy&gt;
-            	&lt;vmCursor /&gt;
-            &lt;/pendingSubscriberPolicy&gt;
-            &lt;pendingDurableSubscriberPolicy&gt;
-                &lt;vmDurableCursor/&gt;
-            &lt;/pendingDurableSubscriberPolicy&gt;
-          &lt;/policyEntry&gt;
-        &lt;/policyEntries&gt;
-      &lt;/policyMap&gt;
-&lt;/destinationPolicy&gt;
-]]></script>
-</div></div>
-<p>Valid Subscriber types are&#160;<strong><em>vmCursor</em></strong>
and <strong><em>fileCursor.</em></strong>&#160;The default is
the store based cursor.<br clear="none">
-Valid Durable Subscriber cursor types are&#160;<strong><em>storeDurableSubscriberCursor</em></strong>,
<strong><em>vmDurableCursor</em></strong> and <strong><em>fileDurableSubscriberCursor.</em></strong>&#160;The
default is the store based cursor</p>
-
-<h4 id="HowdoIconfigureActiveMQtohold100sofmillionsofQueueMessages?-Queues">Queues</h4>
-
-<p>For Queues there is a single dispatch Queue and pending Queue for every destination,
so configuration is slightly different:</p>
-<div class="code panel pdl" style="border-width: 1px;"><div class="codeContent panelContent
pdl">
-<script class="theme: Default; brush: java; gutter: false" type="syntaxhighlighter"><![CDATA[
-&lt;destinationPolicy&gt;
-      &lt;policyMap&gt;
-        &lt;policyEntries&gt;
-          &lt;policyEntry queue=&quot;org.apache.&gt;&quot;&gt;
-            &lt;deadLetterStrategy&gt;
-              &lt;individualDeadLetterStrategy queuePrefix=&quot;Test.DLQ.&quot;/&gt;
-            &lt;/deadLetterStrategy&gt;
-            &lt;pendingQueuePolicy&gt;
-            	&lt;vmQueueCursor /&gt;
-            &lt;/pendingQueuePolicy&gt;
-          &lt;/policyEntry&gt;
-        &lt;/policyEntries&gt;
-      &lt;/policyMap&gt;
- &lt;/destinationPolicy&gt;
-]]></script>
-</div></div>
-<p>Valid Queue cursor types are <strong><em>storeCursor</em></strong>,
<strong><em>vmQueueCursor</em></strong> and <strong><em>fileQueueCursor.</em></strong>&#160;The
default is the store based cursor</p>
-
-<h3 id="HowdoIconfigureActiveMQtohold100sofmillionsofQueueMessages?-SeeAlso">See Also</h3>
-
-<ul><li><a shape="rect" href="producer-flow-control.html">Producer Flow
Control</a></li></ul></div>
+<div class="wiki-content maincontent">Nearly all messaging systems (certainly open
source ones) hold either a copy of a persistent message or a reference to a persisted message
in memory. This is primarily to try and improve performance, but it also can significantly
decrease the complexity of implementation. In fact ActiveMQ version 4 and below worked this
- way - by holding references to persisted messages in memory.<p>However there is a
limitation to this approach, no matter how much memory you have at your disposal, you will
hit a limit to the number persistent messages a broker can handle at any particular time.</p><p>To
get around this limitation, ActiveMQ introduced a paging cache - for all message stores (except
the memory store) to get the best of both worlds - great performance and the ability to hold
100s of millions of messages in persistent store. ActiveMQ is <strong>not</strong>
limited by memory availability, but by the size of the disk available to hold the persisten
 t messages.</p><p>For more information see: <a shape="rect" href="message-cursors.html">Message
Cursors</a></p></div>
         </td>
         <td valign="top">
           <div class="navigation">

Modified: websites/production/activemq/content/scaling-the-depth-of-a-queue.html
==============================================================================
--- websites/production/activemq/content/scaling-the-depth-of-a-queue.html (original)
+++ websites/production/activemq/content/scaling-the-depth-of-a-queue.html Wed Mar 11 10:39:02
2015
@@ -32,16 +32,6 @@
     </style>
     <![endif]-->
 
-          <link href='http://activemq.apache.org/styles/highlighter/styles/shCore.css'
rel='stylesheet' type='text/css' /> 
-      <link href='http://activemq.apache.org/styles/highlighter/styles/shThemeEclipse.css'
rel='stylesheet' type='text/css' /> 
-      <script src='http://activemq.apache.org/styles/highlighter/scripts/shCore.js' type='text/javascript'></script>

-              <script src='http://activemq.apache.org/styles/highlighter/scripts/shBrushJava.js'
type='text/javascript'></script> 
-              <script src='http://activemq.apache.org/styles/highlighter/scripts/shBrushPlain.js'
type='text/javascript'></script> 
-         
-      <script type="text/javascript"> 
-        SyntaxHighlighter.defaults['toolbar'] = false; 
-        SyntaxHighlighter.all(); 
-      </script> 
     
     <title>
     Apache ActiveMQ &#8482; -- Scaling the Depth of a Queue
@@ -82,97 +72,7 @@
   <tbody>
         <tr>
         <td valign="top" width="100%">
-<div class="wiki-content maincontent"><p>Nearly all messaging systems (certainly
open source ones) hold either a copy of a persistent message or a reference to a persisted
message in memory. This is primarily to try and improve performance, but it also can significantly
decrease the complexity of implementation. In fact ActiveMQ version 4 and below worked this
- way - by holding references to persisted messages in memory.</p><p>However there
is a limitation to this approach, no matter how much memory you have at your disposal, you
will hit a limit to the number persistent messages a broker can handle at any particular time.</p><p>To
get around this limitation, ActiveMQ introduced a paging cache - for all message stores (except
the memory store) to get the best of both worlds - great performance and the ability to hold
100s of millions of messages in persistent store. ActiveMQ is <strong>not</strong>
limited by memory availability, but by the size of the disk available to hold the pe
 rsistent messages.</p><p>For more information - see below:</p><p></p><h2
id="ScalingtheDepthofaQueue-MessageCursors">Message Cursors</h2>
-
-<p>A common problem in previous versions of ActiveMQ was <a shape="rect" href="my-producer-blocks.html">running
out of RAM buffer</a> when using non-persistent messaging.</p>
-
-<p>Beginning with ActiveMQ 5.0.0, there is a new memory model that allows messages
to be paged in from storage when space is available (using Store cursors for persistent messages).</p>
-
-<p>Releases prior to 5.0 kept references in memory for all the messages that could
be dispatched to an active Durable Topic Consumer or a Queue. While a reference itself is
not large, it does impose a limit on the maximum number of messages that can be pending delivery.</p>
-
-<p>A typical approach for messaging systems dispatching persistent messages is to pull
them in batches from long term storage when a client is ready to consume them, using a cursor
to maintain the next to dispatch position. This is a robust and very scalable approach, but
not the most performant for cases when the consumer(s) can keep up with the producer(s) of
messages.</p>
-
-<p>ActiveMQ 5.0 takes a hybrid approach, allowing messages to pass from producer to
consumer directly (after the messages have been persisted), but switches back to using cursors
if the consumer(s) fall behind.</p>
-
-<p>When Message Consumers are both active and fast - keeping up with the Message Producer(s)
- messages are stored and then passed to a dispatch queue in the broker associated with the
Consumer: <img class="confluence-embedded-image image-center" src="scaling-the-depth-of-a-queue.data/DispatchFastConsumers.png"
data-image-src="/confluence/download/attachments/48811/DispatchFastConsumers.png?version=1&amp;modificationDate=1193437635000&amp;api=v2"><br
clear="none">
-If a Consumer becomes active after messages are pending from the store for it, or it's slower
than the producer, then messages are paged in to the dispatch queue from a pending cursor:
<img class="confluence-embedded-image image-center" src="scaling-the-depth-of-a-queue.data/DispatchSlowConsumers.png"
data-image-src="/confluence/download/attachments/48811/DispatchSlowConsumers.png?version=1&amp;modificationDate=1193437656000&amp;api=v2"></p>
-
-<h3 id="ScalingtheDepthofaQueue-TypesofCursor">Types of Cursor</h3>
-
-<p>The default message cursor type in ActiveMQ 5.0&#160;is Store based. &#160;It
behaves as above. There are two&#160;additional&#160;types of cursor that could be
used: <strong>VM Cursor</strong> and <strong>File based Cursor</strong>,
described below.</p>
-
-<h4 id="ScalingtheDepthofaQueue-VMCursor">VM Cursor</h4>
-
-<p>The VM Cursor is how ActiveMQ 4.x works: references to a message are held in memory,
and passed to the dispatch queue when needed. This can be very fast, but also has the downside
of not being able to handle very slow consumers or consumers that have been inactive for a
long time: <img class="confluence-embedded-image image-center" src="scaling-the-depth-of-a-queue.data/VMCursor.png"
data-image-src="/confluence/download/attachments/48811/VMCursor.png?version=1&amp;modificationDate=1193439154000&amp;api=v2"></p>
-
-<h4 id="ScalingtheDepthofaQueue-FilebasedCursor">File based Cursor</h4>
-
-<p>The File based Cursor is dervied from the VM Cursor. &#160;When memory in the
broker reaches its limit, it can page messages to temporary files on disk. This type of cursor
can be used when the message store might be relatively slow, but consumers are generally fast.
By buffering to disk, it allows the message broker to handle message bursts from producers
without resorting to paging in from slow storage: <img class="confluence-embedded-image
image-center" src="scaling-the-depth-of-a-queue.data/FileCursor.png" data-image-src="/confluence/download/attachments/48811/FileCursor.png?version=1&amp;modificationDate=1193438877000&amp;api=v2"></p>
-
-<h4 id="ScalingtheDepthofaQueue-PagingforNon-PersistentMessages">Paging for Non-Persistent
Messages</h4>
-
-<p>The store based cursor also handles cursors for non-persistent messages, which are
not stored in the message store. Non-persistent messages are passed directly to the cursor,
so the store based cursor embeds a file based cursor just for these types of messages: <img
class="confluence-embedded-image image-center" src="scaling-the-depth-of-a-queue.data/NonPersistentMsgs.png"
data-image-src="/confluence/download/attachments/48811/NonPersistentMsgs.png?version=1&amp;modificationDate=1193445319000&amp;api=v2"></p>
-
-<h3 id="ScalingtheDepthofaQueue-ConfiguringCursors">Configuring Cursors</h3>
-
-<p>By default, Store based cursors are used, but it is possible to configure different
cursors depending on the destination.</p>
-
-<h4 id="ScalingtheDepthofaQueue-Topicsubscribers">Topic subscribers</h4>
-
-<p>For Topics there is a dispatch queue and pending cursor for every subscriber. &#160;It's
possible to configure different policies for durable subscribers and transient subscribers
- e.g:</p>
-<div class="code panel pdl" style="border-width: 1px;"><div class="codeContent panelContent
pdl">
-<script class="theme: Default; brush: java; gutter: false" type="syntaxhighlighter"><![CDATA[
-&lt;destinationPolicy&gt;
-      &lt;policyMap&gt;
-        &lt;policyEntries&gt;
-          &lt;policyEntry topic=&quot;org.apache.&gt;&quot; producerFlowControl=&quot;false&quot;
memoryLimit=&quot;1mb&quot;&gt;
-            &lt;dispatchPolicy&gt;
-              &lt;strictOrderDispatchPolicy /&gt;
-            &lt;/dispatchPolicy&gt;
-            &lt;deadLetterStrategy&gt;
-              &lt;individualDeadLetterStrategy  topicPrefix=&quot;Test.DLQ.&quot;
/&gt;
-            &lt;/deadLetterStrategy&gt;
-            &lt;pendingSubscriberPolicy&gt;
-            	&lt;vmCursor /&gt;
-            &lt;/pendingSubscriberPolicy&gt;
-            &lt;pendingDurableSubscriberPolicy&gt;
-                &lt;vmDurableCursor/&gt;
-            &lt;/pendingDurableSubscriberPolicy&gt;
-          &lt;/policyEntry&gt;
-        &lt;/policyEntries&gt;
-      &lt;/policyMap&gt;
-&lt;/destinationPolicy&gt;
-]]></script>
-</div></div>
-<p>Valid Subscriber types are&#160;<strong><em>vmCursor</em></strong>
and <strong><em>fileCursor.</em></strong>&#160;The default is
the store based cursor.<br clear="none">
-Valid Durable Subscriber cursor types are&#160;<strong><em>storeDurableSubscriberCursor</em></strong>,
<strong><em>vmDurableCursor</em></strong> and <strong><em>fileDurableSubscriberCursor.</em></strong>&#160;The
default is the store based cursor</p>
-
-<h4 id="ScalingtheDepthofaQueue-Queues">Queues</h4>
-
-<p>For Queues there is a single dispatch Queue and pending Queue for every destination,
so configuration is slightly different:</p>
-<div class="code panel pdl" style="border-width: 1px;"><div class="codeContent panelContent
pdl">
-<script class="theme: Default; brush: java; gutter: false" type="syntaxhighlighter"><![CDATA[
-&lt;destinationPolicy&gt;
-      &lt;policyMap&gt;
-        &lt;policyEntries&gt;
-          &lt;policyEntry queue=&quot;org.apache.&gt;&quot;&gt;
-            &lt;deadLetterStrategy&gt;
-              &lt;individualDeadLetterStrategy queuePrefix=&quot;Test.DLQ.&quot;/&gt;
-            &lt;/deadLetterStrategy&gt;
-            &lt;pendingQueuePolicy&gt;
-            	&lt;vmQueueCursor /&gt;
-            &lt;/pendingQueuePolicy&gt;
-          &lt;/policyEntry&gt;
-        &lt;/policyEntries&gt;
-      &lt;/policyMap&gt;
- &lt;/destinationPolicy&gt;
-]]></script>
-</div></div>
-<p>Valid Queue cursor types are <strong><em>storeCursor</em></strong>,
<strong><em>vmQueueCursor</em></strong> and <strong><em>fileQueueCursor.</em></strong>&#160;The
default is the store based cursor</p>
-
-<h3 id="ScalingtheDepthofaQueue-SeeAlso">See Also</h3>
-
-<ul><li><a shape="rect" href="producer-flow-control.html">Producer Flow
Control</a></li></ul></div>
+<div class="wiki-content maincontent"><p>Nearly all messaging systems (certainly
open source ones) hold either a copy of a persistent message or a reference to a persisted
message in memory. This is primarily to try and improve performance, but it also can significantly
decrease the complexity of implementation. In fact ActiveMQ version 4 and below worked this
- way - by holding references to persisted messages in memory.</p><p>However there
is a limitation to this approach, no matter how much memory you have at your disposal, you
will hit a limit to the number persistent messages a broker can handle at any particular time.</p><p>To
get around this limitation, ActiveMQ introduced a paging cache - for all message stores (except
the memory store) to get the best of both worlds - great performance and the ability to hold
100s of millions of messages in persistent store. ActiveMQ is <strong>not</strong>
limited by memory availability, but by the size of the disk available to hold the pe
 rsistent messages.</p><p>For more information see: <a shape="rect" href="message-cursors.html">Message
Cursors</a></p></div>
         </td>
         <td valign="top">
           <div class="navigation">



Mime
View raw message