activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jose Luna <j-l...@rocketmail.com>
Subject Large number of queues (HowTo)
Date Mon, 18 May 2009 18:27:59 GMT

Hello,

We have the requirement of creating a very large number of queues (tens of thousands) in a
single broker.  My searches turned up several others trying to do something similar, but with
no results. We were recently able to accomplish this, so I decided to write a little howto.
All of this information can be found in the mailing list, wiki, or XML reference, but it wasn't
easy to pull it all together.  Hopefully this will help someone.  Also, please let me know
if there are any inaccuracies, or if anything can be added.

== Thread Count ==
=== "QueueThread" ===
One of the issues that you'll come across is receiving the out of memory error "unable to
create new native thread" when you create thousands of queues.  A "QueueThread" thread is
created for every new queue.  The optimizedDispatch property for the queue policyEntry seemed
to address our needs, the description states "don't use a separate thread for dispatching
from a Queue"  (http://activemq.apache.org/per-destination-policies.html). However, turning
this on just seemed to create "TempQueue" threads instead, still creating a thread for each
queue.  Digging a little deeper, we find that using OptimizedDispatch creates the thread using
a TaskRunner, which should pool the threads.   After further research, we found that the property
org.apache.activemq.UseDedicatedTaskRunner should be set to false so that the task runner
actually pools the threads (http://activemq.apache.org/javalangoutofmemory.html --- this document
also provides other useful info, like
 how to lower the stack size for each thread).   

Example Policy Entry in INSTALL_DIR/conf/activemq.conf:
<policyEntries>
    <policyEntry queue=">" memoryLimit="15mb" optimizedDispatch='true'/>
</policyEntries>

You can either change the 'UseDedicatedTaskRunner' property in your startup script, INSTALL_DIR/bin/activemq:

ACTIVEMQ_OPTS="-Xmx512M -Dorg.apache.activemq.UseDedicatedTaskRunner=false"
Or you can set ACTIVEMQ_OPTS in /etc/activemq.conf .

Note: It would be very useful it the documentation listed above referenced each other, as
they are most useful when used together.

=== "Checkpoint" thread ===
If you're using persistent storage, a "Checkpoint" thread is also created for every queue
that is created. These are used to write persistent messages to the store.  Fortunately this
is already created with a task runner, so setting the UseDedicatedTaskRunner property to false
will prevent these thread from overhwelming your system.

=== Connection related threads ===
If you're also dealing with high thread count due to a very high number of connections, see
 http://fusesource.com/wiki/display/ProdInfo/Understanding+the+Threads+Allocated+in+ActiveMQ.
 Also, consider reducing the stack size for each thread (the JVM option -Xss).

== "too many open files" ==
ActiveMQ uses the amqPersistenceAdapter by default for persistent messages.  Unfortunately,
this persistence adapter (as well as the kahaPersistenceAdapter) opens a file descriptor for
each queue.  When creating large numbers of queues, you'll quickly run into the limit for
your OS.   There are two possible solutions to this:  1.) raise the per-process open file
limit for your OS 2.) Use a jdbc persistent adapter. 
The first can simply be googled for your OS.  The second uses a connection pool to your DB
of choice, and no longer requires an open file for each queue created.

An unrelated issue is running out of file descriptors because there are too many connections.
 (In linux, and other OSes, a file descriptor is used for open sockets).  I don't know of
any other way to handle that, other than increasing the limit for your process.

== Conclusion ==
We were able to create more than 20k queues using ~30 threads in a test environment with a
512MB heap.  Environment: ActiveMQ 5.2.0, SUN Java 1.6.0, Ubuntu 9.04 .  Our testing hasn't
been exhaustive, but everything is running smoothly so far.

JLuna


      

Mime
View raw message