Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (nike.apache.org: domain of lists@nabble.com designates
 216.139.236.158 as permitted sender)
Message-ID: <13737723.post@talk.nabble.com>
Date: Tue, 13 Nov 2007 16:27:05 -0800 (PST)
From: Ken Ringdahl <ken.ringdahl@desktone.com>
To: users@activemq.apache.org
Subject: Re: ActiveMQ Transport Server OutOfMemoryError while running load
 test
In-Reply-To: <13733932.post@talk.nabble.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
References: <13428819.post@talk.nabble.com> <13733932.post@talk.nabble.com>


I should follow up with some of my results.  I did resolve the OOM issue by
setting the thread size (-Xss) in the JVM to 128k.  I was able to ramp up to
6k concurrent connections without any trouble at all.  I was testing
scalability and fail over and was able to fail over these connections to
another ActiveMQ router in ~30 seconds.  I did not push any higher than 6k
since that was far and away more than I would need, but it did seem that
there was significantly more headroom.


Hellweek wrote:
> 
> For the past 2 weeks we have been testing ActiveMQ to answer this question
> ourself.  Here is what we found.
> 
> Creating a Connection takes one thread.
> Creating a Session takes one thread.
> Creating a NON Persisted Topic with a messageListner takes one thread,
> when the topic is no longer used the thread is returned to the system.
> Creating a Persisted Topic with a messageListner takes one thread this
> thread seems to stay around even when the Producer is disposed.
> Creating a QUEUE creates 1 thread per QUEUE when the QUEUE is disposed the
> thread servicing the QUEUE is still in use.
> 
> On Windows 32 Bit we were able to get up to 2,000 threads before we could
> not create anymore.
> On Linux 32 Bit we were able to get up to about 2,500 threads before we
> could not create anymore.  It is important to note that I had to make
> ulimit changes to memory size, virtual memory size and file handles in
> order to reach more then 1,000.
> 
> 
> On Windows 64 Bit with a 64 bit JVM we were able to get up to 5,000
> threads before we could not create anymore.
> On Linux 64 Bit with a 64bit JVM we were able to get up to about 6,000
> threads before we could not create anymore.  It is important to note that
> I had to make ulimit changes to memory size, virtual memory size and file
> handles in order to reach more.
> 
> 
> 
> 
> Ken Ringdahl wrote:
>> 
>> I'm running a load test that intends to throw on the order of several
>> thousand connections at a 2 node broker system with failover (ActiveMQ
>> 4.1.1) configured for tcp transport only.  However, I'm just using a
>> single broker node right now and am getting out of memory errors at about
>> 400 connections.  I've set the Xmx to 1024M and the memory is only ~110
>> MB when  this error occurs in the log.  I've pasted the exceptions below. 
>> The latter 2 exceptions do not immediately follow the first.  They come
>> maybe a minute or so later.  I've tried setting the preFetch to 1 in case
>> it's keeping messages in memory.  But, to be honest, my test doesn't send
>> messages until all of the connections have been established.  Any
>> suggestions as to what might be the problem here?
>> 
>> 
>> 
>> jvm 1    | Exception in thread "ActiveMQ Transport Server:
>> tcp://localhost:61616" java.lang.OutOfMemoryError: unable to create new
>> native thread
>> jvm 1    |      at java.lang.Thread.start0(Native Method)
>> jvm 1    |      at java.lang.Thread.start(Thread.java:574)
>> jvm 1    |      at
>> org.apache.activemq.thread.DedicatedTaskRunner.<init>(DedicatedTaskRunner.java:45)
>> jvm 1    |      at
>> org.apache.activemq.thread.TaskRunnerFactory.createTaskRunner(TaskRunnerFactory.java:77)
>> jvm 1    |      at
>> org.apache.activemq.broker.TransportConnection.<init>(TransportConnection.java:174)
>> jvm 1    |      at
>> org.apache.activemq.broker.jmx.ManagedTransportConnection.<init>(ManagedTransportConnection.java:55)
>> jvm 1    |      at
>> org.apache.activemq.broker.jmx.ManagedTransportConnector.createConnection(ManagedTransportConnector.java:56)
>> jvm 1    |      at
>> org.apache.activemq.broker.TransportConnector$1.onAccept(TransportConnector.java:147)
>> jvm 1    |      at
>> org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:167)
>> jvm 1    |      at java.lang.Thread.run(Thread.java:595)
>> jvm 1    | Exception in thread "RMI RenewClean-[172.16.105.110:36883]"
>> java.lang.OutOfMemoryError: unable to create new native thread
>> jvm 1    |      at java.lang.Thread.start0(Native Method)
>> jvm 1    |      at java.lang.Thread.start(Thread.java:574)
>> jvm 1    |      at
>> sun.rmi.transport.tcp.TCPChannel.free(TCPChannel.java:321)
>> jvm 1    |      at sun.rmi.server.UnicastRef.free(UnicastRef.java:395)
>> jvm 1    |      at sun.rmi.server.UnicastRef.done(UnicastRef.java:412)
>> jvm 1    |      at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
>> jvm 1    |      at
>> sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:328)
>> jvm 1    |      at
>> sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:144)
>> jvm 1    |      at
>> sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:539)
>> jvm 1    |      at java.lang.Thread.run(Thread.java:595)
>> jvm 1    | Exception in thread "ActiveMQ Journal Checkpoint Worker"
>> java.lang.OutOfMemoryError: unable to create new native thread
>> jvm 1    |      at java.lang.Thread.start0(Native Method)
>> jvm 1    |      at java.lang.Thread.start(Thread.java:574)
>> jvm 1    |      at
>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:429)
>> jvm 1    |      at
>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:857)
>> jvm 1    |      at
>> org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:376)
>> jvm 1    |      at
>> org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129)
>> jvm 1    |      at
>> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101)
>> jvm 1    |      at
>> org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
>> jvm 1    |      at
>> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
>> 
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/ActiveMQ-Transport-Server-OutOfMemoryError-while-running-load-test-tf4697790s2354.html#a13737723
Sent from the ActiveMQ - User mailing list archive at Nabble.com.