activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <gary.tu...@gmail.com>
Subject Re: Out of Memory on 5.3
Date Mon, 09 Nov 2009 14:39:30 GMT
Looks like AMQ-2468 does not resolve this as there is also an issue
with the message list. I have opened
https://issues.apache.org/activemq/browse/AMQ-2481 to track this and
already have a reproducible test case.
Thanks for the heads up.

2009/11/9 afei <afei1689@126.com>:
>
>
> when a large of messages in queue,and no consumer or the consumer is very
> slow, the OOM problem occur, because :
> in org.apache.activemq.broker.region.Queue,the 588 line is :
>  doBrowse(true, browsedMessages, this.getMaxExpirePageSize());
> ,transform to :
> doBrowse(false, browsedMessages, this.getMaxExpirePageSize());
>  is ok.
>
>
> Dejan Bosanac wrote:
>>
>> Hi Mitch,
>>
>> yeah, I said in thread I was referring to, that it is working with
>> "regular"
>> stomp connector. I started investigating AMQ-2440 patch the other day,
>> should be have something soon.
>>
>> Cheers
>> --
>> Dejan Bosanac - http://twitter.com/dejanb
>>
>> Open Source Integration - http://fusesource.com/
>> ActiveMQ in Action - http://www.manning.com/snyder/
>> Blog - http://www.nighttale.net
>>
>>
>> On Wed, Oct 28, 2009 at 6:18 PM, Mitch Granger
>> <mitch.granger@sophos.com>wrote:
>>
>>> So we turned off stomp+nio and went back to plain old stomp and so far
>>> it's
>>> working fine.  New(IO) isn't always better, I guess :-)
>>>
>>> Seems like maybe it's this issue ->
>>> https://issues.apache.org/activemq/browse/AMQ-2440
>>>
>>>
>>> afei wrote:
>>>
>>>> i have same problem
>>>>
>>>> http://www.nabble.com/file/p26093204/aaaaaa.jpg aaaaaa.jpg
>>>>
>>>>
>>>> themitchy wrote:
>>>>
>>>>> This is what we've done to tune so far:
>>>>>
>>>>>  - UseDedicatedTaskRunner=false
>>>>>  - flow control is off
>>>>>  - stomp transport uses transport.closeAsync=false
>>>>>
>>>>> I agree that it is because of the high number of open/close connections
>>>>> from Stomp.  When we monitor through JConsole we can see more threads
>>>>> starting up for each new connection.  The problem is that these threads
>>>>> don't get let go.  Even though the stomp clients are disconnecting the
>>>>> number of threads that get released is less than the number created.
>>>>> So
>>>>>  the thread count goes up and up until it fails.  All of the above
>>>>> settings/tuning only delay when it will hit the wall.
>>>>>
>>>>> Dejan Bosanac wrote:
>>>>>
>>>>>> Hi Mitch,
>>>>>>
>>>>>> I think the root cause of this problem is that you probably have
Stomp
>>>>>> clients that open/close connection at a high rate. I simulated this
>>>>>> problem
>>>>>> on OSX with a StompLoadTest (
>>>>>>
>>>>>> http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/transport/stomp/StompLoadTest.java?view=log
>>>>>> ),
>>>>>> while trying to reproduce "too many open files" problem. You can
find
>>>>>> some
>>>>>> of my findings (and workaround) in this thread.
>>>>>>
>>>>>>
>>>>>> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tt25888831.html#a26010080
>>>>>>
>>>>>> (BTW. it is producing "too many open files" problem on linux)
>>>>>> Basically,
>>>>>> the
>>>>>> problem with stomp is that every send is done in separate connection
>>>>>> and
>>>>>> thus considered to be a new producer for every message. So when
>>>>>> producer
>>>>>> flow control is hit, the producers are piling up and probably not
>>>>>> releasing
>>>>>> connections. Thus you can observe large number of tcp connections
on
>>>>>> the
>>>>>> system in state TIME_WAIT (and TIME_CLOSE), which causes that the
>>>>>> system
>>>>>> limit is hit at one point. In the above thread, you can find a
>>>>>> workaround
>>>>>> that worked for me for that test. I started investigating this more
>>>>>> and
>>>>>> hopefully I'll have some more findings in the near future.
>>>>>>
>>>>>> Cheers
>>>>>> --
>>>>>> Dejan Bosanac - http://twitter.com/dejanb
>>>>>>
>>>>>> Open Source Integration - http://fusesource.com/
>>>>>> ActiveMQ in Action - http://www.manning.com/snyder/
>>>>>> Blog - http://www.nighttale.net
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 27, 2009 at 1:07 AM, Mitch Granger
>>>>>> <mitch.granger@sophos.com>wrote:
>>>>>>
>>>>>>  Update: We've [nearly] proven that this only happens with AMQ running
>>>>>>> on
>>>>>>> openVZ.  What exactly is causing it, we're still not sure.  After
>>>>>>> memoryUsage is met, the number of threads skyrockets until we
get
>>>>>>> OutOfMemoryError.
>>>>>>>
>>>>>>> It works just fine on regular hardware; We're going to try VMWare
>>>>>>> tomorrow.
>>>>>>>
>>>>>>> One thing really worth mentioning is that by using the fileCursor
we
>>>>>>> actually started seeing it use the Temp Store.  When reading
about
>>>>>>> systemUsage it is NOT intuitive that the Temp Store does not
come
>>>>>>> into
>>>>>>> play
>>>>>>> with the default cursor.  Anyone keeping a significant volume
of
>>>>>>> messages on
>>>>>>> their queues should be well served by changing the cursor.
>>>>>>>
>>>>>>>
>>>>>>> Mitch Granger wrote:
>>>>>>>
>>>>>>>  Config is attached.  We have also tried the activemq-scalability.xml
>>>>>>>> with
>>>>>>>> the only change being adding a stomp connector.
>>>>>>>>
>>>>>>>> Once we hit the memoryUsage limit we can [sometimes] connect
new
>>>>>>>> consumers
>>>>>>>> but nothing comes back after we send the SUBSCRIBE frame.
>>>>>>>>
>>>>>>>> I expect sending to fail when we hit this limit but if we
can't
>>>>>>>> subscribe
>>>>>>>> there's no chance of recovering from this state.
>>>>>>>>
>>>>>>>> Rob Davies wrote:
>>>>>>>>
>>>>>>>>  On 26 Oct 2009, at 17:38, themitchy wrote:
>>>>>>>>>
>>>>>>>>>  We're using only persistent messages and heap size
is set to 2GB
>>>>>>>>> yet
>>>>>>>>> we
>>>>>>>>>
>>>>>>>>>> hit
>>>>>>>>>> the memoryUsage limit quite quickly (system usage
config below).
>>>>>>>>>> This
>>>>>>>>>> is
>>>>>>>>>> followed by "java.lang.OutOfMemoryError: unable to
create new
>>>>>>>>>> native
>>>>>>>>>>  thread"
>>>>>>>>>> as the process quickly reaches the 2GB of heap we
gave it.  How
>>>>>>>>>> are
>>>>>>>>>> we
>>>>>>>>>> getting to that point with the memoryUsage limit
set far below it?
>>>>>>>>>>
>>>>>>>>>> Is there no way to get AMQ to gracefully limit it's
memory usage?
>>>>>>>>>>
>>>>>>>>>>      <systemUsage>
>>>>>>>>>>          <systemUsage>
>>>>>>>>>>              <memoryUsage>
>>>>>>>>>>                  <memoryUsage limit="256
mb"/>
>>>>>>>>>>              </memoryUsage>
>>>>>>>>>>              <storeUsage>
>>>>>>>>>>                  <storeUsage limit="60
gb" name="foo"/>
>>>>>>>>>>              </storeUsage>
>>>>>>>>>>              <tempUsage>
>>>>>>>>>>                  <tempUsage limit="60
gb"/>
>>>>>>>>>>              </tempUsage>
>>>>>>>>>>          </systemUsage>
>>>>>>>>>>      </systemUsage>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> View this message in context:
>>>>>>>>>> http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26064098.html
>>>>>>>>>> Sent from the ActiveMQ - User mailing list archive
at Nabble.com.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  Can you send the rest of your config ?
>>>>>>>>>
>>>>>>>>> Rob Davies
>>>>>>>>> http://twitter.com/rajdavies
>>>>>>>>> I work here: http://fusesource.com
>>>>>>>>> My Blog: http://rajdavies.blogspot.com/
>>>>>>>>> I'm writing this: http://www.manning.com/snyder/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>
>>>>
>>
>>
>> -----
>> Dejan Bosanac
>>
>> Open Source Integration - http://fusesource.com/
>> ActiveMQ in Action - http://www.manning.com/snyder/
>> Blog - http://www.nighttale.net
>>
> http://old.nabble.com/file/p26264779/dump.jpg
> --
> View this message in context: http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26264779.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>



-- 
http://blog.garytully.com

Open Source Integration
http://fusesource.com

Mime
View raw message