Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (athena.apache.org: domain of lists@nabble.com designates
 216.139.236.158 as permitted sender)
Message-ID: <23682395.post@talk.nabble.com>
Date: Sat, 23 May 2009 01:29:38 -0700 (PDT)
From: DataMover <david.pirzadeh@cision.com>
To: users@activemq.apache.org
Subject: Re: Large number of queues (HowTo)
In-Reply-To: <911381.8914.qm@web50804.mail.re2.yahoo.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
References: <911381.8914.qm@web50804.mail.re2.yahoo.com>


Any chance we can get a hold of your test code.
1. If I can test against a known configuration that is known to work I can
find what I'm doing wrong faster
2. I see that most most posts about working setups are NOT Centos, in fact I
have seen two posts that use Ubuntu successfully. Am I to change my linux
distro?
3. How many is "many" connections or machines that connect to the server? At
more than fifty we see issues.
4. How would having a total of 256k ram effect this?


Jose Luna-2 wrote:
> 
> 
> Hello,
> 
> We have the requirement of creating a very large number of queues (tens of
> thousands) in a single broker.  My searches turned up several others
> trying to do something similar, but with no results. We were recently able
> to accomplish this, so I decided to write a little howto. All of this
> information can be found in the mailing list, wiki, or XML reference, but
> it wasn't easy to pull it all together.  Hopefully this will help someone. 
> Also, please let me know if there are any inaccuracies, or if anything can
> be added.
> 
> == Thread Count ==
> === "QueueThread" ===
> One of the issues that you'll come across is receiving the out of memory
> error "unable to create new native thread" when you create thousands of
> queues.  A "QueueThread" thread is created for every new queue.  The
> optimizedDispatch property for the queue policyEntry seemed to address our
> needs, the description states "don't use a separate thread for dispatching
> from a Queue"  (http://activemq.apache.org/per-destination-policies.html).
> However, turning this on just seemed to create "TempQueue" threads
> instead, still creating a thread for each queue.  Digging a little deeper,
> we find that using OptimizedDispatch creates the thread using a
> TaskRunner, which should pool the threads.   After further research, we
> found that the property org.apache.activemq.UseDedicatedTaskRunner should
> be set to false so that the task runner actually pools the threads
> (http://activemq.apache.org/javalangoutofmemory.html --- this document
> also provides other useful info, like
>  how to lower the stack size for each thread).   
> 
> Example Policy Entry in INSTALL_DIR/conf/activemq.conf:
> <policyEntries>
>     <policyEntry queue=">" memoryLimit="15mb" optimizedDispatch='true'/>
> </policyEntries>
> 
> You can either change the 'UseDedicatedTaskRunner' property in your
> startup script, INSTALL_DIR/bin/activemq: 
> ACTIVEMQ_OPTS="-Xmx512M
> -Dorg.apache.activemq.UseDedicatedTaskRunner=false"
> Or you can set ACTIVEMQ_OPTS in /etc/activemq.conf .
> 
> Note: It would be very useful it the documentation listed above referenced
> each other, as they are most useful when used together.
> 
> === "Checkpoint" thread ===
> If you're using persistent storage, a "Checkpoint" thread is also created
> for every queue that is created. These are used to write persistent
> messages to the store.  Fortunately this is already created with a task
> runner, so setting the UseDedicatedTaskRunner property to false will
> prevent these thread from overhwelming your system.
> 
> === Connection related threads ===
> If you're also dealing with high thread count due to a very high number of
> connections, see 
> http://fusesource.com/wiki/display/ProdInfo/Understanding+the+Threads+Allocated+in+ActiveMQ. 
> Also, consider reducing the stack size for each thread (the JVM option
> -Xss).
> 
> == "too many open files" ==
> ActiveMQ uses the amqPersistenceAdapter by default for persistent
> messages.  Unfortunately, this persistence adapter (as well as the
> kahaPersistenceAdapter) opens a file descriptor for each queue.  When
> creating large numbers of queues, you'll quickly run into the limit for
> your OS.   There are two possible solutions to this:  1.) raise the
> per-process open file limit for your OS 2.) Use a jdbc persistent adapter. 
> The first can simply be googled for your OS.  The second uses a connection
> pool to your DB of choice, and no longer requires an open file for each
> queue created.
> 
> An unrelated issue is running out of file descriptors because there are
> too many connections.  (In linux, and other OSes, a file descriptor is
> used for open sockets).  I don't know of any other way to handle that,
> other than increasing the limit for your process.
> 
> == Conclusion ==
> We were able to create more than 20k queues using ~30 threads in a test
> environment with a 512MB heap.  Environment: ActiveMQ 5.2.0, SUN Java
> 1.6.0, Ubuntu 9.04 .  Our testing hasn't been exhaustive, but everything
> is running smoothly so far.
> 
> JLuna
> 
> 
>       
> 
> 

-- 
View this message in context: http://www.nabble.com/Large-number-of-queues-%28HowTo%29-tp23602928p23682395.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.