Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (herse.apache.org: domain of lists@nabble.com designates
 72.21.53.35 as permitted sender)
Message-ID: <10160068.post@talk.nabble.com>
Date: Tue, 24 Apr 2007 05:34:32 -0700 (PDT)
From: Ale Sarco <ale@sarco.com.ar>
To: users@activemq.apache.org
Subject: Re: 3.1 broker OutofMemoryError : unable to create new native
 thread
In-Reply-To: <10145361.post@talk.nabble.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
References: <9844430.post@talk.nabble.com>
 <ec6e67fd0704060131o5430b4c4t5b874cd92eafca6@mail.gmail.com>
 <10032556.post@talk.nabble.com>
 <ec6e67fd0704170330l5f3ef13bq36632816ffe3866b@mail.gmail.com>
 <10136935.post@talk.nabble.com> <10145361.post@talk.nabble.com>


Replying myself:

Setting wireFormat.maxInactivityDuration=0 in the producers, solved the
problem of connections not being closed. However, I think this is a bug,
because the connection is not closed due to an exception being thrown by
something not related to the connection itself. I didn't see the source
code, but feels like the closing connection code not being inside a finally
block. 

Now, what I don't understand is why a TCP connection is created every time.
This happened no matter if I used ActiveMQ's PooledConnectionFactory or
Spring's SingleConnectionFactory. If I send 10000 messages at once, it will
use the same connection for all of them, but if there's an interval between
sending 2 messages, it will use 2 different connections. 

Any ideas?
Alex


Ale Sarco wrote:
> 
> Further investigation led to find this one, by setting the log to DEBUG.
> This happens when a new message is about to be received:
> 
> 2007-04-23 18:16:05,966 [ActiveMQ Transport: tcp:///127.0.0.1:54165] DEBUG
> org.apache.activemq.broker.region.AbstractRegion  - Removing consumer:
> ID:magellan-15043-1177347682999-1:9:-1:1
> 2007-04-23 18:16:05,967 [ActiveMQ Transport: tcp:///127.0.0.1:54165] DEBUG
> org.apache.activemq.broker.TransportConnection  - Stopping connection:
> /127.0.0.1:54165
> 2007-04-23 18:16:05,968 [ActiveMQ Transport: tcp:///127.0.0.1:54165] DEBUG
> org.apache.activemq.broker.TransportConnection  - Could not stop
> transport: java.lang.ClassCastException:
> org.apache.activemq.transport.InactivityMonitor$1
> java.lang.ClassCastException:
> org.apache.activemq.transport.InactivityMonitor$1
>         at
> edu.emory.mathcs.backport.java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.compareTo(ScheduledThreadPoolExecutor.java:189)
>         at
> edu.emory.mathcs.backport.java.util.PriorityQueue.remove(PriorityQueue.java:507)
>         at
> edu.emory.mathcs.backport.java.util.concurrent.DelayQueue.remove(DelayQueue.java:379)
>         ...
> 
> 2007-04-23 18:16:05,969 [ActiveMQ Transport Server: tcp://localhost:6002]
> DEBUG org.apache.activemq.transport.WireFormatNegotiator  - Sending:
> WireFormatInfo { version=2, properties={TightEncodingEnabled=true,
> CacheSize=1024, TcpNoDelayEnabled=true, SizePrefixDisabled=false,
> StackTraceEnabled=true, MaxInactivityDuration=30000, CacheEnabled=true},
> magic=[A,c,t,i,v,e,M,Q]}
> 2007-04-23 18:16:05,970 [ActiveMQ Transport: tcp:///127.0.0.1:54166] DEBUG
> org.apache.activemq.transport.WireFormatNegotiator  - Received WireFormat:
> WireFormatInfo { version=2, properties={TightEncodingEnabled=true,
> CacheSize=1024, TcpNoDelayEnabled=true, SizePrefixDisabled=false,
> StackTraceEnabled=true, MaxInactivityDuration=30000, CacheEnabled=true},
> magic=[A,c,t,i,v,e,M,Q]}
> 2007-04-23 18:16:05,970 [ActiveMQ Transport: tcp:///127.0.0.1:54166] DEBUG
> org.apache.activemq.transport.WireFormatNegotiator  -
> tcp:///127.0.0.1:54166 before negotiation: OpenWireFormat{version=2,
> cacheEnabled=false, stackTraceEnabled=false, tightEncodingEnabled=false,
> sizePrefixDisabled=false}
> 2007-04-23 18:16:05,970 [ActiveMQ Transport: tcp:///127.0.0.1:54166] DEBUG
> org.apache.activemq.transport.WireFormatNegotiator  -
> tcp:///127.0.0.1:54166 after negotiation: OpenWireFormat{version=2,
> cacheEnabled=true, stackTraceEnabled=true, tightEncodingEnabled=true,
> sizePrefixDisabled=false}
> 2007-04-23 18:16:05,970 [ActiveMQ Transport: tcp:///127.0.0.1:54166] DEBUG
> org.apache.activemq.broker.TransportConnection  - Setting up new
> connection:
> org.apache.activemq.broker.jmx.ManagedTransportConnection@b36b28
> 2007-04-23 18:16:06,011 [ActiveMQ Transport: tcp:///127.0.0.1:54166] DEBUG
> org.apache.activemq.broker.region.AbstractRegion  - Adding consumer:
> ID:magellan-15043-1177347682999-1:10:-1:1
> 
> So, for some reason, the old connection to the server cannot be closed
> because of that ClassCastException (might this be due to an
> InactivityMonitor instance being transferred with the message, and then
> being tried to be deserialized with a different classloader, as the
> producer and consumer are in 2 different webapps inside the same
> container?), and then another connection is created. Eventually, no more
> connections will be available.
> Why is the connection being tried to be closed in the first place, instead
> of being reused? Can this be avoided by setting
> wireFormat.maxInactivityDuration=0 in the broker? What would be the impact
> of doing this?
> 
> Thanks!
> Alex.
> 
> James.Strachan wrote:
>> 
>> On 4/17/07, Ale Sarco <ale@sarco.com.ar> wrote:
>>>
>>> Hi James,
>>>
>>> Using ActiveMQ 4.1.1 with Spring I experienced this "unable to create
>>> new
>>> native thread" exception in one of my production servers last night. The
>>> thing is that server does not has a large messaging traffic, it just
>>> send 2
>>> or 3 messages every couple of minutes to another webapp running in the
>>> same
>>> Tomcat, and this is only for 2 hours a day, and there were plenty of
>>> memory
>>> available on the heap for that VM.
>>> I agree that is probably related to the file descriptors that you
>>> mentioned,
>>> because when I tried to log in to that server via SSH I got an error
>>> about
>>> something like "insufficient resources". Only when I restarted the
>>> webapp I
>>> was able to login again.
>>> Now, what file descriptors is this about? Is a (logical) file created
>>> somewhere every time a message is sent? If so, how do I avoid this
>>> problem,
>>> I guess they should be reused or something?
>> 
>> File descriptors are often used for various things in the OS; files,
>> sockets, sometimes threads etc. ActiveMQ does not create a file or
>> socket per message :).
>> 
>> BTW how do things look in jconsole; do you have lots of connections on
>> your broker? I wonder if you are not closing connections after they
>> have been used (you are really meant to only use a small number of
>> connections and lots of sessions).
>> http://activemq.apache.org/how-do-i-use-jms-efficiently.html
>> 
>> Just up the file descriptors setting on your OS and it should fix this
>> 
>> 
>> -- 
>> 
>> James
>> -------
>> http://radio.weblogs.com/0112098/
>> 
>> 
> 
> 


-- 
View this message in context: http://www.nabble.com/3.1-broker-OutofMemoryError-%3A-unable-to-create-new-native-thread-tf3527977s2354.html#a10160068
Sent from the ActiveMQ - User mailing list archive at Nabble.com.