activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mitch Granger <mitch.gran...@sophos.com>
Subject Re: Out of Memory on 5.3
Date Wed, 28 Oct 2009 17:18:19 GMT
So we turned off stomp+nio and went back to plain old stomp and so far 
it's working fine.  New(IO) isn't always better, I guess :-)

Seems like maybe it's this issue -> 
https://issues.apache.org/activemq/browse/AMQ-2440

afei wrote:
> i have same problem 
> 
> 
> http://www.nabble.com/file/p26093204/aaaaaa.jpg aaaaaa.jpg 
> 
> 
> 
> themitchy wrote:
>> This is what we've done to tune so far:
>>
>>   - UseDedicatedTaskRunner=false
>>   - flow control is off
>>   - stomp transport uses transport.closeAsync=false
>>
>> I agree that it is because of the high number of open/close connections 
>> from Stomp.  When we monitor through JConsole we can see more threads 
>> starting up for each new connection.  The problem is that these threads 
>> don't get let go.  Even though the stomp clients are disconnecting the 
>> number of threads that get released is less than the number created.  So 
>>   the thread count goes up and up until it fails.  All of the above 
>> settings/tuning only delay when it will hit the wall.
>>
>> Dejan Bosanac wrote:
>>> Hi Mitch,
>>>
>>> I think the root cause of this problem is that you probably have Stomp
>>> clients that open/close connection at a high rate. I simulated this
>>> problem
>>> on OSX with a StompLoadTest (
>>> http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/transport/stomp/StompLoadTest.java?view=log),
>>> while trying to reproduce "too many open files" problem. You can find
>>> some
>>> of my findings (and workaround) in this thread.
>>>
>>> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tt25888831.html#a26010080
>>>
>>> (BTW. it is producing "too many open files" problem on linux) Basically,
>>> the
>>> problem with stomp is that every send is done in separate connection and
>>> thus considered to be a new producer for every message. So when producer
>>> flow control is hit, the producers are piling up and probably not
>>> releasing
>>> connections. Thus you can observe large number of tcp connections on the
>>> system in state TIME_WAIT (and TIME_CLOSE), which causes that the system
>>> limit is hit at one point. In the above thread, you can find a workaround
>>> that worked for me for that test. I started investigating this more and
>>> hopefully I'll have some more findings in the near future.
>>>
>>> Cheers
>>> --
>>> Dejan Bosanac - http://twitter.com/dejanb
>>>
>>> Open Source Integration - http://fusesource.com/
>>> ActiveMQ in Action - http://www.manning.com/snyder/
>>> Blog - http://www.nighttale.net
>>>
>>>
>>> On Tue, Oct 27, 2009 at 1:07 AM, Mitch Granger
>>> <mitch.granger@sophos.com>wrote:
>>>
>>>> Update: We've [nearly] proven that this only happens with AMQ running on
>>>> openVZ.  What exactly is causing it, we're still not sure.  After
>>>> memoryUsage is met, the number of threads skyrockets until we get
>>>> OutOfMemoryError.
>>>>
>>>> It works just fine on regular hardware; We're going to try VMWare
>>>> tomorrow.
>>>>
>>>> One thing really worth mentioning is that by using the fileCursor we
>>>> actually started seeing it use the Temp Store.  When reading about
>>>> systemUsage it is NOT intuitive that the Temp Store does not come into
>>>> play
>>>> with the default cursor.  Anyone keeping a significant volume of
>>>> messages on
>>>> their queues should be well served by changing the cursor.
>>>>
>>>>
>>>> Mitch Granger wrote:
>>>>
>>>>> Config is attached.  We have also tried the activemq-scalability.xml
>>>>> with
>>>>> the only change being adding a stomp connector.
>>>>>
>>>>> Once we hit the memoryUsage limit we can [sometimes] connect new
>>>>> consumers
>>>>> but nothing comes back after we send the SUBSCRIBE frame.
>>>>>
>>>>> I expect sending to fail when we hit this limit but if we can't
>>>>> subscribe
>>>>> there's no chance of recovering from this state.
>>>>>
>>>>> Rob Davies wrote:
>>>>>
>>>>>> On 26 Oct 2009, at 17:38, themitchy wrote:
>>>>>>
>>>>>>  We're using only persistent messages and heap size is set to 2GB
yet 
>>>>>> we
>>>>>>> hit
>>>>>>> the memoryUsage limit quite quickly (system usage config below).

>>>>>>> This
>>>>>>> is
>>>>>>> followed by "java.lang.OutOfMemoryError: unable to create new
native
>>>>>>>  thread"
>>>>>>> as the process quickly reaches the 2GB of heap we gave it.  How
are
>>>>>>> we
>>>>>>> getting to that point with the memoryUsage limit set far below
it?
>>>>>>>
>>>>>>> Is there no way to get AMQ to gracefully limit it's memory usage?
>>>>>>>
>>>>>>>       <systemUsage>
>>>>>>>           <systemUsage>
>>>>>>>               <memoryUsage>
>>>>>>>                   <memoryUsage limit="256 mb"/>
>>>>>>>               </memoryUsage>
>>>>>>>               <storeUsage>
>>>>>>>                   <storeUsage limit="60 gb" name="foo"/>
>>>>>>>               </storeUsage>
>>>>>>>               <tempUsage>
>>>>>>>                   <tempUsage limit="60 gb"/>
>>>>>>>               </tempUsage>
>>>>>>>           </systemUsage>
>>>>>>>       </systemUsage>
>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26064098.html
>>>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>> Can you send the rest of your config ?
>>>>>>
>>>>>> Rob Davies
>>>>>> http://twitter.com/rajdavies
>>>>>> I work here: http://fusesource.com
>>>>>> My Blog: http://rajdavies.blogspot.com/
>>>>>> I'm writing this: http://www.manning.com/snyder/
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>
> 

Mime
View raw message