activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From afei <afei1...@126.com>
Subject Re: Out of Memory on 5.3
Date Wed, 28 Oct 2009 11:43:12 GMT

i have same problem 


http://www.nabble.com/file/p26093204/aaaaaa.jpg aaaaaa.jpg 



themitchy wrote:
> 
> This is what we've done to tune so far:
> 
>   - UseDedicatedTaskRunner=false
>   - flow control is off
>   - stomp transport uses transport.closeAsync=false
> 
> I agree that it is because of the high number of open/close connections 
> from Stomp.  When we monitor through JConsole we can see more threads 
> starting up for each new connection.  The problem is that these threads 
> don't get let go.  Even though the stomp clients are disconnecting the 
> number of threads that get released is less than the number created.  So 
>   the thread count goes up and up until it fails.  All of the above 
> settings/tuning only delay when it will hit the wall.
> 
> Dejan Bosanac wrote:
>> Hi Mitch,
>> 
>> I think the root cause of this problem is that you probably have Stomp
>> clients that open/close connection at a high rate. I simulated this
>> problem
>> on OSX with a StompLoadTest (
>> http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/transport/stomp/StompLoadTest.java?view=log),
>> while trying to reproduce "too many open files" problem. You can find
>> some
>> of my findings (and workaround) in this thread.
>> 
>> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tt25888831.html#a26010080
>> 
>> (BTW. it is producing "too many open files" problem on linux) Basically,
>> the
>> problem with stomp is that every send is done in separate connection and
>> thus considered to be a new producer for every message. So when producer
>> flow control is hit, the producers are piling up and probably not
>> releasing
>> connections. Thus you can observe large number of tcp connections on the
>> system in state TIME_WAIT (and TIME_CLOSE), which causes that the system
>> limit is hit at one point. In the above thread, you can find a workaround
>> that worked for me for that test. I started investigating this more and
>> hopefully I'll have some more findings in the near future.
>> 
>> Cheers
>> --
>> Dejan Bosanac - http://twitter.com/dejanb
>> 
>> Open Source Integration - http://fusesource.com/
>> ActiveMQ in Action - http://www.manning.com/snyder/
>> Blog - http://www.nighttale.net
>> 
>> 
>> On Tue, Oct 27, 2009 at 1:07 AM, Mitch Granger
>> <mitch.granger@sophos.com>wrote:
>> 
>>> Update: We've [nearly] proven that this only happens with AMQ running on
>>> openVZ.  What exactly is causing it, we're still not sure.  After
>>> memoryUsage is met, the number of threads skyrockets until we get
>>> OutOfMemoryError.
>>>
>>> It works just fine on regular hardware; We're going to try VMWare
>>> tomorrow.
>>>
>>> One thing really worth mentioning is that by using the fileCursor we
>>> actually started seeing it use the Temp Store.  When reading about
>>> systemUsage it is NOT intuitive that the Temp Store does not come into
>>> play
>>> with the default cursor.  Anyone keeping a significant volume of
>>> messages on
>>> their queues should be well served by changing the cursor.
>>>
>>>
>>> Mitch Granger wrote:
>>>
>>>> Config is attached.  We have also tried the activemq-scalability.xml
>>>> with
>>>> the only change being adding a stomp connector.
>>>>
>>>> Once we hit the memoryUsage limit we can [sometimes] connect new
>>>> consumers
>>>> but nothing comes back after we send the SUBSCRIBE frame.
>>>>
>>>> I expect sending to fail when we hit this limit but if we can't
>>>> subscribe
>>>> there's no chance of recovering from this state.
>>>>
>>>> Rob Davies wrote:
>>>>
>>>>> On 26 Oct 2009, at 17:38, themitchy wrote:
>>>>>
>>>>>  We're using only persistent messages and heap size is set to 2GB yet

>>>>> we
>>>>>> hit
>>>>>> the memoryUsage limit quite quickly (system usage config below).

>>>>>> This
>>>>>> is
>>>>>> followed by "java.lang.OutOfMemoryError: unable to create new native
>>>>>>  thread"
>>>>>> as the process quickly reaches the 2GB of heap we gave it.  How are
>>>>>> we
>>>>>> getting to that point with the memoryUsage limit set far below it?
>>>>>>
>>>>>> Is there no way to get AMQ to gracefully limit it's memory usage?
>>>>>>
>>>>>>       <systemUsage>
>>>>>>           <systemUsage>
>>>>>>               <memoryUsage>
>>>>>>                   <memoryUsage limit="256 mb"/>
>>>>>>               </memoryUsage>
>>>>>>               <storeUsage>
>>>>>>                   <storeUsage limit="60 gb" name="foo"/>
>>>>>>               </storeUsage>
>>>>>>               <tempUsage>
>>>>>>                   <tempUsage limit="60 gb"/>
>>>>>>               </tempUsage>
>>>>>>           </systemUsage>
>>>>>>       </systemUsage>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26064098.html
>>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>> Can you send the rest of your config ?
>>>>>
>>>>> Rob Davies
>>>>> http://twitter.com/rajdavies
>>>>> I work here: http://fusesource.com
>>>>> My Blog: http://rajdavies.blogspot.com/
>>>>> I'm writing this: http://www.manning.com/snyder/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
> 
> 

-- 
View this message in context: http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26093204.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message