activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From afei <afei1...@126.com>
Subject Re: Out of Memory on 5.3
Date Wed, 11 Nov 2009 07:11:59 GMT

when consuming a large of messages,the method:asyncWakeup() is invoked
crazily,so the executor
has a great deal of  runnable that callback Queue.iterate(), but 
Queue.iterate() is much slower than the increasing of runnable in the
executor. this result in OOM.

avoid to invoke the method:asyncWakeup()  frequently???


Gary Tully wrote:
> 
> iterate actually does a dispatch. when consumers change etc, we again
> try and dispatch to ensure prefetch buffers are maintained.
> 
> Do a svn up to r834579 as I committed a fix to trunk for this today
> resolving https://issues.apache.org/activemq/browse/AMQ-2481
> 
> 2009/11/10 afei <afei1689@126.com>:
>>
>> in org.apache.activemq.broker.region ,why so many invoke asyncWakeup(),
>> what is the method:iterate() doing?
>>
>>
>> afei wrote:
>>>
>>> in addition,another problem of OOM.
>>>
>>>
>>>
>>> Gary Tully wrote:
>>>>
>>>> fyi: you can disable periodic message expiry processing using a
>>>> destination policy entry that sets expireMessagesPeriod = 0
>>>>
>>>> 2009/11/9 afei <afei1689@126.com>:
>>>>>
>>>>>
>>>>> when a large of messages in queue,and no consumer or the consumer is
>>>>> very
>>>>> slow, the OOM problem occur, because :
>>>>> in org.apache.activemq.broker.region.Queue,the 588 line is :
>>>>>  doBrowse(true, browsedMessages, this.getMaxExpirePageSize());
>>>>> ,transform to :
>>>>> doBrowse(false, browsedMessages, this.getMaxExpirePageSize());
>>>>>  is ok.
>>>>>
>>>>>
>>>>> Dejan Bosanac wrote:
>>>>>>
>>>>>> Hi Mitch,
>>>>>>
>>>>>> yeah, I said in thread I was referring to, that it is working with
>>>>>> "regular"
>>>>>> stomp connector. I started investigating AMQ-2440 patch the other
>>>>>> day,
>>>>>> should be have something soon.
>>>>>>
>>>>>> Cheers
>>>>>> --
>>>>>> Dejan Bosanac - http://twitter.com/dejanb
>>>>>>
>>>>>> Open Source Integration - http://fusesource.com/
>>>>>> ActiveMQ in Action - http://www.manning.com/snyder/
>>>>>> Blog - http://www.nighttale.net
>>>>>>
>>>>>>
>>>>>> On Wed, Oct 28, 2009 at 6:18 PM, Mitch Granger
>>>>>> <mitch.granger@sophos.com>wrote:
>>>>>>
>>>>>>> So we turned off stomp+nio and went back to plain old stomp and
so
>>>>>>> far
>>>>>>> it's
>>>>>>> working fine.  New(IO) isn't always better, I guess :-)
>>>>>>>
>>>>>>> Seems like maybe it's this issue ->
>>>>>>> https://issues.apache.org/activemq/browse/AMQ-2440
>>>>>>>
>>>>>>>
>>>>>>> afei wrote:
>>>>>>>
>>>>>>>> i have same problem
>>>>>>>>
>>>>>>>> http://www.nabble.com/file/p26093204/aaaaaa.jpg aaaaaa.jpg
>>>>>>>>
>>>>>>>>
>>>>>>>> themitchy wrote:
>>>>>>>>
>>>>>>>>> This is what we've done to tune so far:
>>>>>>>>>
>>>>>>>>>  - UseDedicatedTaskRunner=false
>>>>>>>>>  - flow control is off
>>>>>>>>>  - stomp transport uses transport.closeAsync=false
>>>>>>>>>
>>>>>>>>> I agree that it is because of the high number of open/close
>>>>>>>>> connections
>>>>>>>>> from Stomp.  When we monitor through JConsole we can
see more
>>>>>>>>> threads
>>>>>>>>> starting up for each new connection.  The problem is
that these
>>>>>>>>> threads
>>>>>>>>> don't get let go.  Even though the stomp clients are
disconnecting
>>>>>>>>> the
>>>>>>>>> number of threads that get released is less than the
number
>>>>>>>>> created.
>>>>>>>>> So
>>>>>>>>>  the thread count goes up and up until it fails.  All
of the above
>>>>>>>>> settings/tuning only delay when it will hit the wall.
>>>>>>>>>
>>>>>>>>> Dejan Bosanac wrote:
>>>>>>>>>
>>>>>>>>>> Hi Mitch,
>>>>>>>>>>
>>>>>>>>>> I think the root cause of this problem is that you
probably have
>>>>>>>>>> Stomp
>>>>>>>>>> clients that open/close connection at a high rate.
I simulated
>>>>>>>>>> this
>>>>>>>>>> problem
>>>>>>>>>> on OSX with a StompLoadTest (
>>>>>>>>>>
>>>>>>>>>> http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/transport/stomp/StompLoadTest.java?view=log
>>>>>>>>>> ),
>>>>>>>>>> while trying to reproduce "too many open files" problem.
You can
>>>>>>>>>> find
>>>>>>>>>> some
>>>>>>>>>> of my findings (and workaround) in this thread.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tt25888831.html#a26010080
>>>>>>>>>>
>>>>>>>>>> (BTW. it is producing "too many open files" problem
on linux)
>>>>>>>>>> Basically,
>>>>>>>>>> the
>>>>>>>>>> problem with stomp is that every send is done in
separate
>>>>>>>>>> connection
>>>>>>>>>> and
>>>>>>>>>> thus considered to be a new producer for every message.
So when
>>>>>>>>>> producer
>>>>>>>>>> flow control is hit, the producers are piling up
and probably not
>>>>>>>>>> releasing
>>>>>>>>>> connections. Thus you can observe large number of
tcp connections
>>>>>>>>>> on
>>>>>>>>>> the
>>>>>>>>>> system in state TIME_WAIT (and TIME_CLOSE), which
causes that the
>>>>>>>>>> system
>>>>>>>>>> limit is hit at one point. In the above thread, you
can find a
>>>>>>>>>> workaround
>>>>>>>>>> that worked for me for that test. I started investigating
this
>>>>>>>>>> more
>>>>>>>>>> and
>>>>>>>>>> hopefully I'll have some more findings in the near
future.
>>>>>>>>>>
>>>>>>>>>> Cheers
>>>>>>>>>> --
>>>>>>>>>> Dejan Bosanac - http://twitter.com/dejanb
>>>>>>>>>>
>>>>>>>>>> Open Source Integration - http://fusesource.com/
>>>>>>>>>> ActiveMQ in Action - http://www.manning.com/snyder/
>>>>>>>>>> Blog - http://www.nighttale.net
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Oct 27, 2009 at 1:07 AM, Mitch Granger
>>>>>>>>>> <mitch.granger@sophos.com>wrote:
>>>>>>>>>>
>>>>>>>>>>  Update: We've [nearly] proven that this only happens
with AMQ
>>>>>>>>>> running
>>>>>>>>>>> on
>>>>>>>>>>> openVZ.  What exactly is causing it, we're still
not sure.
>>>>>>>>>>>  After
>>>>>>>>>>> memoryUsage is met, the number of threads skyrockets
until we
>>>>>>>>>>> get
>>>>>>>>>>> OutOfMemoryError.
>>>>>>>>>>>
>>>>>>>>>>> It works just fine on regular hardware; We're
going to try
>>>>>>>>>>> VMWare
>>>>>>>>>>> tomorrow.
>>>>>>>>>>>
>>>>>>>>>>> One thing really worth mentioning is that by
using the
>>>>>>>>>>> fileCursor
>>>>>>>>>>> we
>>>>>>>>>>> actually started seeing it use the Temp Store.
 When reading
>>>>>>>>>>> about
>>>>>>>>>>> systemUsage it is NOT intuitive that the Temp
Store does not
>>>>>>>>>>> come
>>>>>>>>>>> into
>>>>>>>>>>> play
>>>>>>>>>>> with the default cursor.  Anyone keeping a significant
volume of
>>>>>>>>>>> messages on
>>>>>>>>>>> their queues should be well served by changing
the cursor.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Mitch Granger wrote:
>>>>>>>>>>>
>>>>>>>>>>>  Config is attached.  We have also tried the
>>>>>>>>>>> activemq-scalability.xml
>>>>>>>>>>>> with
>>>>>>>>>>>> the only change being adding a stomp connector.
>>>>>>>>>>>>
>>>>>>>>>>>> Once we hit the memoryUsage limit we can
[sometimes] connect
>>>>>>>>>>>> new
>>>>>>>>>>>> consumers
>>>>>>>>>>>> but nothing comes back after we send the
SUBSCRIBE frame.
>>>>>>>>>>>>
>>>>>>>>>>>> I expect sending to fail when we hit this
limit but if we can't
>>>>>>>>>>>> subscribe
>>>>>>>>>>>> there's no chance of recovering from this
state.
>>>>>>>>>>>>
>>>>>>>>>>>> Rob Davies wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>  On 26 Oct 2009, at 17:38, themitchy wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>  We're using only persistent messages
and heap size is set to
>>>>>>>>>>>>> 2GB
>>>>>>>>>>>>> yet
>>>>>>>>>>>>> we
>>>>>>>>>>>>>
>>>>>>>>>>>>>> hit
>>>>>>>>>>>>>> the memoryUsage limit quite quickly
(system usage config
>>>>>>>>>>>>>> below).
>>>>>>>>>>>>>> This
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> followed by "java.lang.OutOfMemoryError:
unable to create new
>>>>>>>>>>>>>> native
>>>>>>>>>>>>>>  thread"
>>>>>>>>>>>>>> as the process quickly reaches the
2GB of heap we gave it.
>>>>>>>>>>>>>>  How
>>>>>>>>>>>>>> are
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>> getting to that point with the memoryUsage
limit set far
>>>>>>>>>>>>>> below
>>>>>>>>>>>>>> it?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is there no way to get AMQ to gracefully
limit it's memory
>>>>>>>>>>>>>> usage?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>      <systemUsage>
>>>>>>>>>>>>>>          <systemUsage>
>>>>>>>>>>>>>>              <memoryUsage>
>>>>>>>>>>>>>>                  <memoryUsage
limit="256 mb"/>
>>>>>>>>>>>>>>              </memoryUsage>
>>>>>>>>>>>>>>              <storeUsage>
>>>>>>>>>>>>>>                  <storeUsage
limit="60 gb" name="foo"/>
>>>>>>>>>>>>>>              </storeUsage>
>>>>>>>>>>>>>>              <tempUsage>
>>>>>>>>>>>>>>                  <tempUsage
limit="60 gb"/>
>>>>>>>>>>>>>>              </tempUsage>
>>>>>>>>>>>>>>          </systemUsage>
>>>>>>>>>>>>>>      </systemUsage>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> View this message in context:
>>>>>>>>>>>>>> http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26064098.html
>>>>>>>>>>>>>> Sent from the ActiveMQ - User mailing
list archive at
>>>>>>>>>>>>>> Nabble.com.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  Can you send the rest of your config
?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Rob Davies
>>>>>>>>>>>>> http://twitter.com/rajdavies
>>>>>>>>>>>>> I work here: http://fusesource.com
>>>>>>>>>>>>> My Blog: http://rajdavies.blogspot.com/
>>>>>>>>>>>>> I'm writing this: http://www.manning.com/snyder/
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> -----
>>>>>> Dejan Bosanac
>>>>>>
>>>>>> Open Source Integration - http://fusesource.com/
>>>>>> ActiveMQ in Action - http://www.manning.com/snyder/
>>>>>> Blog - http://www.nighttale.net
>>>>>>
>>>>> http://old.nabble.com/file/p26264779/dump.jpg
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26264779.html
>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> http://blog.garytully.com
>>>>
>>>> Open Source Integration
>>>> http://fusesource.com
>>>>
>>>>
>>>  http://old.nabble.com/file/p26278228/oom.jpg
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26279671.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
> 
> 
> 
> -- 
> http://blog.garytully.com
> 
> Open Source Integration
> http://fusesource.com
> 
> 

-- 
View this message in context: http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26297020.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message