activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dejan Bosanac <de...@nighttale.net>
Subject Re: Out of Memory on 5.3
Date Wed, 28 Oct 2009 17:33:00 GMT
Hi Mitch,

yeah, I said in thread I was referring to, that it is working with "regular"
stomp connector. I started investigating AMQ-2440 patch the other day,
should be have something soon.

Cheers
--
Dejan Bosanac - http://twitter.com/dejanb

Open Source Integration - http://fusesource.com/
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net


On Wed, Oct 28, 2009 at 6:18 PM, Mitch Granger <mitch.granger@sophos.com>wrote:

> So we turned off stomp+nio and went back to plain old stomp and so far it's
> working fine.  New(IO) isn't always better, I guess :-)
>
> Seems like maybe it's this issue ->
> https://issues.apache.org/activemq/browse/AMQ-2440
>
>
> afei wrote:
>
>> i have same problem
>>
>> http://www.nabble.com/file/p26093204/aaaaaa.jpg aaaaaa.jpg
>>
>>
>> themitchy wrote:
>>
>>> This is what we've done to tune so far:
>>>
>>>  - UseDedicatedTaskRunner=false
>>>  - flow control is off
>>>  - stomp transport uses transport.closeAsync=false
>>>
>>> I agree that it is because of the high number of open/close connections
>>> from Stomp.  When we monitor through JConsole we can see more threads
>>> starting up for each new connection.  The problem is that these threads
>>> don't get let go.  Even though the stomp clients are disconnecting the
>>> number of threads that get released is less than the number created.  So
>>>  the thread count goes up and up until it fails.  All of the above
>>> settings/tuning only delay when it will hit the wall.
>>>
>>> Dejan Bosanac wrote:
>>>
>>>> Hi Mitch,
>>>>
>>>> I think the root cause of this problem is that you probably have Stomp
>>>> clients that open/close connection at a high rate. I simulated this
>>>> problem
>>>> on OSX with a StompLoadTest (
>>>>
>>>> http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/transport/stomp/StompLoadTest.java?view=log
>>>> ),
>>>> while trying to reproduce "too many open files" problem. You can find
>>>> some
>>>> of my findings (and workaround) in this thread.
>>>>
>>>>
>>>> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tt25888831.html#a26010080
>>>>
>>>> (BTW. it is producing "too many open files" problem on linux) Basically,
>>>> the
>>>> problem with stomp is that every send is done in separate connection and
>>>> thus considered to be a new producer for every message. So when producer
>>>> flow control is hit, the producers are piling up and probably not
>>>> releasing
>>>> connections. Thus you can observe large number of tcp connections on the
>>>> system in state TIME_WAIT (and TIME_CLOSE), which causes that the system
>>>> limit is hit at one point. In the above thread, you can find a
>>>> workaround
>>>> that worked for me for that test. I started investigating this more and
>>>> hopefully I'll have some more findings in the near future.
>>>>
>>>> Cheers
>>>> --
>>>> Dejan Bosanac - http://twitter.com/dejanb
>>>>
>>>> Open Source Integration - http://fusesource.com/
>>>> ActiveMQ in Action - http://www.manning.com/snyder/
>>>> Blog - http://www.nighttale.net
>>>>
>>>>
>>>> On Tue, Oct 27, 2009 at 1:07 AM, Mitch Granger
>>>> <mitch.granger@sophos.com>wrote:
>>>>
>>>>  Update: We've [nearly] proven that this only happens with AMQ running
>>>>> on
>>>>> openVZ.  What exactly is causing it, we're still not sure.  After
>>>>> memoryUsage is met, the number of threads skyrockets until we get
>>>>> OutOfMemoryError.
>>>>>
>>>>> It works just fine on regular hardware; We're going to try VMWare
>>>>> tomorrow.
>>>>>
>>>>> One thing really worth mentioning is that by using the fileCursor we
>>>>> actually started seeing it use the Temp Store.  When reading about
>>>>> systemUsage it is NOT intuitive that the Temp Store does not come into
>>>>> play
>>>>> with the default cursor.  Anyone keeping a significant volume of
>>>>> messages on
>>>>> their queues should be well served by changing the cursor.
>>>>>
>>>>>
>>>>> Mitch Granger wrote:
>>>>>
>>>>>  Config is attached.  We have also tried the activemq-scalability.xml
>>>>>> with
>>>>>> the only change being adding a stomp connector.
>>>>>>
>>>>>> Once we hit the memoryUsage limit we can [sometimes] connect new
>>>>>> consumers
>>>>>> but nothing comes back after we send the SUBSCRIBE frame.
>>>>>>
>>>>>> I expect sending to fail when we hit this limit but if we can't
>>>>>> subscribe
>>>>>> there's no chance of recovering from this state.
>>>>>>
>>>>>> Rob Davies wrote:
>>>>>>
>>>>>>  On 26 Oct 2009, at 17:38, themitchy wrote:
>>>>>>>
>>>>>>>  We're using only persistent messages and heap size is set to
2GB yet
>>>>>>> we
>>>>>>>
>>>>>>>> hit
>>>>>>>> the memoryUsage limit quite quickly (system usage config
below).
>>>>>>>> This
>>>>>>>> is
>>>>>>>> followed by "java.lang.OutOfMemoryError: unable to create
new native
>>>>>>>>  thread"
>>>>>>>> as the process quickly reaches the 2GB of heap we gave it.
 How are
>>>>>>>> we
>>>>>>>> getting to that point with the memoryUsage limit set far
below it?
>>>>>>>>
>>>>>>>> Is there no way to get AMQ to gracefully limit it's memory
usage?
>>>>>>>>
>>>>>>>>      <systemUsage>
>>>>>>>>          <systemUsage>
>>>>>>>>              <memoryUsage>
>>>>>>>>                  <memoryUsage limit="256 mb"/>
>>>>>>>>              </memoryUsage>
>>>>>>>>              <storeUsage>
>>>>>>>>                  <storeUsage limit="60 gb" name="foo"/>
>>>>>>>>              </storeUsage>
>>>>>>>>              <tempUsage>
>>>>>>>>                  <tempUsage limit="60 gb"/>
>>>>>>>>              </tempUsage>
>>>>>>>>          </systemUsage>
>>>>>>>>      </systemUsage>
>>>>>>>>
>>>>>>>> --
>>>>>>>> View this message in context:
>>>>>>>> http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26064098.html
>>>>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>>>>
>>>>>>>>
>>>>>>>>  Can you send the rest of your config ?
>>>>>>>
>>>>>>> Rob Davies
>>>>>>> http://twitter.com/rajdavies
>>>>>>> I work here: http://fusesource.com
>>>>>>> My Blog: http://rajdavies.blogspot.com/
>>>>>>> I'm writing this: http://www.manning.com/snyder/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message