activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: Force Transaction data to spool to disk
Date Sat, 10 Mar 2018 03:38:22 GMT
As I understand it, the cursor spools out to disk messages that have been
committed into the memory store. If they're participating in a transaction,
they're not committed and are instead just sitting in memory in the
org.apache.activemq.store.memory.MemoryTransactionStore. (I assume we're
talking about non-persistent messages here, since persistent ones are
written to a disk-based store to begin with, so spooling via a cursor isn't
a factor there.) If you look at the code for
org.apache.activemq.store.memory.MemoryTransactionStore.Tx, you'll see that
the messages are stored in a List<AddMessageCommand>, which means they're
simply stored on the heap and not eligible to be spooled automatically.

Here are a few things you could do:

   1. You could modify
   org.apache.activemq.store.memory.MemoryTransactionStore.Tx to do your own
   spooling, perhaps to a KahaDBStore objecct you would create for that
   purpose.
   2. You could increase the size of the broker's heap. 1GB is nothing by
   today's standards, especially if you're expecting to be using more than
   that from time to time. Throwing more heap space at the problem could be a
   perfectly viable option if you're able to place an upper limit on how much
   heap might be used by a single transaction and then size the broker much
   larger than that.
   3. If you're not able to get enough RAM to handle the heap size you'd
   need, you might try allocating (at the operating system) some swap space to
   be used as virtual memory, and then setting the -Xmx argument to a large
   enough value. I can't say for sure that this will work (I'm not sure the
   JVM will be willing to allocate a heap that spans both RAM and swap), but
   you could easily test by setting -Xms to the same value as -Xmx. But be
   aware that the OS may be considerably less performant when using swap
   (because disk is much slower than RAM) and that Linux in particular may not
   choose to stop using swap even if it gets back to a state where it could
   fit everything in RAM, so there are significant potential disadvantages to
   this approach even if it works.
   4. Get your producers to send fewer messages in a transaction.
   5. Get your producers to send smaller messages in a transaction (e.g. by
   storing the content externally, such as in a cache, and then just passing
   the key rather than the full content).

Hope this help.

Tim

On Fri, Mar 9, 2018 at 12:23 PM, bmadaras <bryan.madaras@armssoftware.com>
wrote:

> I have been able to successfully have large number of persistent messages
> spool over to the filesystem exactly as intended by using the
> cursorMemoryHighWaterMark for messages that have been enqueued to the
> queue.
>
> The issue I am running into, it happens only a few times per day in our
> system, is that a large transaction is created that tries to enqueue a
> couple hundred thousand messages and during the transaction none of the
> rules in place seem to cause the messages to get spooled over to disk.  If
> the number of messages is large enough the broker can still run out of
> memory.
>
> For this scenario I am not too worried about speed as much as I am about
> keeping the broker alive to receive and store the messages.
>
> I am using the file based cursor as I had noticed during some larger
> transactions that was causing the queue to hit the
> cursorMemoryHighWaterMark
> messages that had been committed to the queue were getting blocked behind
> the larger transaction which does not seem to occur with the file based
> cursor.
>
> Any thoughts on how to handle this scenario would be greatly appreciated.
>
> The broker is started with a gig of heap space.  -Xmx1G
>                 <destinationPolicy>
>                         <policyMap>
>                                 <policyEntries>
>
>                                         <policyEntry topic=">"
> producerFlowControl="true">
>
> <pendingMessageLimitStrategy>
>
> <constantPendingMessageLimitStrategy
>
> limit="1000"
> />
>
> </pendingMessageLimitStrategy>
>                                         </policyEntry>
>                                         <policyEntry queue=">"
> producerFlowControl="false" cursorMemoryHighWaterMark="10">
>
>
>
> <pendingQueuePolicy><fileQueueCursor/></pendingQueuePolicy>
>
>                                         </policyEntry>
>                                 </policyEntries>
>                         </policyMap>
>                 </destinationPolicy>
>
>                 <systemUsage>
>                         <systemUsage>
>                                 <memoryUsage>
>                                         <memoryUsage limit="64 mb" />
>                                 </memoryUsage>
>                                 <storeUsage>
>                                         <storeUsage limit="30 gb" />
>                                </storeUsage>
>                                 <tempUsage>
>                                         <tempUsage limit="2 gb" />
>                                 </tempUsage>
>                         </systemUsage>
>                 </systemUsage>
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message