activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From afei <afei1...@126.com>
Subject Re: Out of Memory on 5.3
Date Fri, 13 Nov 2009 02:21:20 GMT

the test scene: after accumulate a large of messages(about 5 millions) in
queue,begin to consume it.

i do a modifications,look at the attach (
http://old.nabble.com/file/p26328726/Queue.java Queue.java ),it look like
ok.
the test xml configuration.


Gary Tully wrote:
> 
> https://issues.apache.org/activemq/browse/AMQ-2483 is tracking this.Could
> you attach your test case and/or configuration to the jira issue. thx. It
> should be possible to reuse an existing task for some of the iterations,
> but
> it is important that no wakeup request is ignored so that the the
> destination keeps responding.
> 
> 2009/11/12 afei <afei1689@126.com>
> 
>>
>> when consuming a large of messages,the method:asyncWakeup() is invoked
>> crazily,so the executor
>> has a great deal of  runnable that callback Queue.iterate(), but
>> Queue.iterate() is much slower than the increasing of runnable in the
>> executor. this result in OOM.
>> the picture of memory dump:
>>
>> avoid to invoke the method:asyncWakeup()  frequently???
>>
>>
>> Gary Tully wrote:
>> >
>> > iterate actually does a dispatch. when consumers change etc, we again
>> > try and dispatch to ensure prefetch buffers are maintained.
>> >
>> > Do a svn up to r834579 as I committed a fix to trunk for this today
>> > resolving https://issues.apache.org/activemq/browse/AMQ-2481
>> >
>> > 2009/11/10 afei <afei1689@126.com>:
>> >>
>> >> in org.apache.activemq.broker.region ,why so many invoke
>> asyncWakeup(),
>> >> what is the method:iterate() doing?
>> >>
>> >>
>> >> afei wrote:
>> >>>
>> >>> in addition,another problem of OOM.
>> >>>
>> >>>
>> >>>
>> >>> Gary Tully wrote:
>> >>>>
>> >>>> fyi: you can disable periodic message expiry processing using a
>> >>>> destination policy entry that sets expireMessagesPeriod = 0
>> >>>>
>> >>>> 2009/11/9 afei <afei1689@126.com>:
>> >>>>>
>> >>>>>
>> >>>>> when a large of messages in queue,and no consumer or the consumer
>> is
>> >>>>> very
>> >>>>> slow, the OOM problem occur, because :
>> >>>>> in org.apache.activemq.broker.region.Queue,the 588 line is :
>> >>>>>  doBrowse(true, browsedMessages, this.getMaxExpirePageSize());
>> >>>>> ,transform to :
>> >>>>> doBrowse(false, browsedMessages, this.getMaxExpirePageSize());
>> >>>>>  is ok.
>> >>>>>
>> >>>>>
>> >>>>> Dejan Bosanac wrote:
>> >>>>>>
>> >>>>>> Hi Mitch,
>> >>>>>>
>> >>>>>> yeah, I said in thread I was referring to, that it is working
with
>> >>>>>> "regular"
>> >>>>>> stomp connector. I started investigating AMQ-2440 patch
the other
>> >>>>>> day,
>> >>>>>> should be have something soon.
>> >>>>>>
>> >>>>>> Cheers
>> >>>>>> --
>> >>>>>> Dejan Bosanac - http://twitter.com/dejanb
>> >>>>>>
>> >>>>>> Open Source Integration - http://fusesource.com/
>> >>>>>> ActiveMQ in Action - http://www.manning.com/snyder/
>> >>>>>> Blog - http://www.nighttale.net
>> >>>>>>
>> >>>>>>
>> >>>>>> On Wed, Oct 28, 2009 at 6:18 PM, Mitch Granger
>> >>>>>> <mitch.granger@sophos.com>wrote:
>> >>>>>>
>> >>>>>>> So we turned off stomp+nio and went back to plain old
stomp and
>> so
>> >>>>>>> far
>> >>>>>>> it's
>> >>>>>>> working fine.  New(IO) isn't always better, I guess
:-)
>> >>>>>>>
>> >>>>>>> Seems like maybe it's this issue ->
>> >>>>>>> https://issues.apache.org/activemq/browse/AMQ-2440
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> afei wrote:
>> >>>>>>>
>> >>>>>>>> i have same problem
>> >>>>>>>>
>> >>>>>>>> http://www.nabble.com/file/p26093204/aaaaaa.jpg
aaaaaa.jpg
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> themitchy wrote:
>> >>>>>>>>
>> >>>>>>>>> This is what we've done to tune so far:
>> >>>>>>>>>
>> >>>>>>>>>  - UseDedicatedTaskRunner=false
>> >>>>>>>>>  - flow control is off
>> >>>>>>>>>  - stomp transport uses transport.closeAsync=false
>> >>>>>>>>>
>> >>>>>>>>> I agree that it is because of the high number
of open/close
>> >>>>>>>>> connections
>> >>>>>>>>> from Stomp.  When we monitor through JConsole
we can see more
>> >>>>>>>>> threads
>> >>>>>>>>> starting up for each new connection.  The problem
is that these
>> >>>>>>>>> threads
>> >>>>>>>>> don't get let go.  Even though the stomp clients
are
>> disconnecting
>> >>>>>>>>> the
>> >>>>>>>>> number of threads that get released is less
than the number
>> >>>>>>>>> created.
>> >>>>>>>>> So
>> >>>>>>>>>  the thread count goes up and up until it fails.
 All of the
>> above
>> >>>>>>>>> settings/tuning only delay when it will hit
the wall.
>> >>>>>>>>>
>> >>>>>>>>> Dejan Bosanac wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Hi Mitch,
>> >>>>>>>>>>
>> >>>>>>>>>> I think the root cause of this problem is
that you probably
>> have
>> >>>>>>>>>> Stomp
>> >>>>>>>>>> clients that open/close connection at a
high rate. I simulated
>> >>>>>>>>>> this
>> >>>>>>>>>> problem
>> >>>>>>>>>> on OSX with a StompLoadTest (
>> >>>>>>>>>>
>> >>>>>>>>>>
>> http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/transport/stomp/StompLoadTest.java?view=log
>> >>>>>>>>>> ),
>> >>>>>>>>>> while trying to reproduce "too many open
files" problem. You
>> can
>> >>>>>>>>>> find
>> >>>>>>>>>> some
>> >>>>>>>>>> of my findings (and workaround) in this
thread.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tt25888831.html#a26010080
>> >>>>>>>>>>
>> >>>>>>>>>> (BTW. it is producing "too many open files"
problem on linux)
>> >>>>>>>>>> Basically,
>> >>>>>>>>>> the
>> >>>>>>>>>> problem with stomp is that every send is
done in separate
>> >>>>>>>>>> connection
>> >>>>>>>>>> and
>> >>>>>>>>>> thus considered to be a new producer for
every message. So
>> when
>> >>>>>>>>>> producer
>> >>>>>>>>>> flow control is hit, the producers are piling
up and probably
>> not
>> >>>>>>>>>> releasing
>> >>>>>>>>>> connections. Thus you can observe large
number of tcp
>> connections
>> >>>>>>>>>> on
>> >>>>>>>>>> the
>> >>>>>>>>>> system in state TIME_WAIT (and TIME_CLOSE),
which causes that
>> the
>> >>>>>>>>>> system
>> >>>>>>>>>> limit is hit at one point. In the above
thread, you can find a
>> >>>>>>>>>> workaround
>> >>>>>>>>>> that worked for me for that test. I started
investigating this
>> >>>>>>>>>> more
>> >>>>>>>>>> and
>> >>>>>>>>>> hopefully I'll have some more findings in
the near future.
>> >>>>>>>>>>
>> >>>>>>>>>> Cheers
>> >>>>>>>>>> --
>> >>>>>>>>>> Dejan Bosanac - http://twitter.com/dejanb
>> >>>>>>>>>>
>> >>>>>>>>>> Open Source Integration - http://fusesource.com/
>> >>>>>>>>>> ActiveMQ in Action - http://www.manning.com/snyder/
>> >>>>>>>>>> Blog - http://www.nighttale.net
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Tue, Oct 27, 2009 at 1:07 AM, Mitch Granger
>> >>>>>>>>>> <mitch.granger@sophos.com>wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>  Update: We've [nearly] proven that this
only happens with AMQ
>> >>>>>>>>>> running
>> >>>>>>>>>>> on
>> >>>>>>>>>>> openVZ.  What exactly is causing it,
we're still not sure.
>> >>>>>>>>>>>  After
>> >>>>>>>>>>> memoryUsage is met, the number of threads
skyrockets until we
>> >>>>>>>>>>> get
>> >>>>>>>>>>> OutOfMemoryError.
>> >>>>>>>>>>>
>> >>>>>>>>>>> It works just fine on regular hardware;
We're going to try
>> >>>>>>>>>>> VMWare
>> >>>>>>>>>>> tomorrow.
>> >>>>>>>>>>>
>> >>>>>>>>>>> One thing really worth mentioning is
that by using the
>> >>>>>>>>>>> fileCursor
>> >>>>>>>>>>> we
>> >>>>>>>>>>> actually started seeing it use the Temp
Store.  When reading
>> >>>>>>>>>>> about
>> >>>>>>>>>>> systemUsage it is NOT intuitive that
the Temp Store does not
>> >>>>>>>>>>> come
>> >>>>>>>>>>> into
>> >>>>>>>>>>> play
>> >>>>>>>>>>> with the default cursor.  Anyone keeping
a significant volume
>> of
>> >>>>>>>>>>> messages on
>> >>>>>>>>>>> their queues should be well served by
changing the cursor.
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> Mitch Granger wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>  Config is attached.  We have also tried
the
>> >>>>>>>>>>> activemq-scalability.xml
>> >>>>>>>>>>>> with
>> >>>>>>>>>>>> the only change being adding a stomp
connector.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Once we hit the memoryUsage limit
we can [sometimes] connect
>> >>>>>>>>>>>> new
>> >>>>>>>>>>>> consumers
>> >>>>>>>>>>>> but nothing comes back after we
send the SUBSCRIBE frame.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> I expect sending to fail when we
hit this limit but if we
>> can't
>> >>>>>>>>>>>> subscribe
>> >>>>>>>>>>>> there's no chance of recovering
from this state.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Rob Davies wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>  On 26 Oct 2009, at 17:38, themitchy
wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>  We're using only persistent
messages and heap size is set
>> to
>> >>>>>>>>>>>>> 2GB
>> >>>>>>>>>>>>> yet
>> >>>>>>>>>>>>> we
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>> hit
>> >>>>>>>>>>>>>> the memoryUsage limit quite
quickly (system usage config
>> >>>>>>>>>>>>>> below).
>> >>>>>>>>>>>>>> This
>> >>>>>>>>>>>>>> is
>> >>>>>>>>>>>>>> followed by "java.lang.OutOfMemoryError:
unable to create
>> new
>> >>>>>>>>>>>>>> native
>> >>>>>>>>>>>>>>  thread"
>> >>>>>>>>>>>>>> as the process quickly reaches
the 2GB of heap we gave it.
>> >>>>>>>>>>>>>>  How
>> >>>>>>>>>>>>>> are
>> >>>>>>>>>>>>>> we
>> >>>>>>>>>>>>>> getting to that point with
the memoryUsage limit set far
>> >>>>>>>>>>>>>> below
>> >>>>>>>>>>>>>> it?
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Is there no way to get AMQ
to gracefully limit it's memory
>> >>>>>>>>>>>>>> usage?
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>      <systemUsage>
>> >>>>>>>>>>>>>>          <systemUsage>
>> >>>>>>>>>>>>>>              <memoryUsage>
>> >>>>>>>>>>>>>>                  <memoryUsage
limit="256 mb"/>
>> >>>>>>>>>>>>>>              </memoryUsage>
>> >>>>>>>>>>>>>>              <storeUsage>
>> >>>>>>>>>>>>>>                  <storeUsage
limit="60 gb" name="foo"/>
>> >>>>>>>>>>>>>>              </storeUsage>
>> >>>>>>>>>>>>>>              <tempUsage>
>> >>>>>>>>>>>>>>                  <tempUsage
limit="60 gb"/>
>> >>>>>>>>>>>>>>              </tempUsage>
>> >>>>>>>>>>>>>>          </systemUsage>
>> >>>>>>>>>>>>>>      </systemUsage>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> --
>> >>>>>>>>>>>>>> View this message in context:
>> >>>>>>>>>>>>>>
>> http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26064098.html
>> >>>>>>>>>>>>>> Sent from the ActiveMQ -
User mailing list archive at
>> >>>>>>>>>>>>>> Nabble.com.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>  Can you send the rest of
your config ?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Rob Davies
>> >>>>>>>>>>>>> http://twitter.com/rajdavies
>> >>>>>>>>>>>>> I work here: http://fusesource.com
>> >>>>>>>>>>>>> My Blog: http://rajdavies.blogspot.com/
>> >>>>>>>>>>>>> I'm writing this: http://www.manning.com/snyder/
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> -----
>> >>>>>> Dejan Bosanac
>> >>>>>>
>> >>>>>> Open Source Integration - http://fusesource.com/
>> >>>>>> ActiveMQ in Action - http://www.manning.com/snyder/
>> >>>>>> Blog - http://www.nighttale.net
>> >>>>>>
>> >>>>> http://old.nabble.com/file/p26264779/dump.jpg
>> >>>>> --
>> >>>>> View this message in context:
>> >>>>> http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26264779.html
>> >>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> http://blog.garytully.com
>> >>>>
>> >>>> Open Source Integration
>> >>>> http://fusesource.com
>> >>>>
>> >>>>
>> >>>  http://old.nabble.com/file/p26278228/oom.jpg
>> >>>
>> >>
>> >> --
>> >> View this message in context:
>> >> http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26279671.html
>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >>
>> >>
>> >
>> >
>> >
>> > --
>> > http://blog.garytully.com
>> >
>> > Open Source Integration
>> > http://fusesource.com
>> >
>> >
>> http://old.nabble.com/file/p26312669/oom.jpg
>> --
>> View this message in context:
>> http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26312669.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
> 
> 
> -- 
> http://blog.garytully.com
> 
> Open Source Integration
> http://fusesource.com
> 
> 
http://old.nabble.com/file/p26328726/activemq.xml activemq.xml 
http://old.nabble.com/file/p26328726/activemq.xml activemq.xml 
-- 
View this message in context: http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26328726.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message