activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <gary.tu...@gmail.com>
Subject Re: Out of Memory on 5.3
Date Thu, 12 Nov 2009 09:51:08 GMT
https://issues.apache.org/activemq/browse/AMQ-2483 is tracking this.Could
you attach your test case and/or configuration to the jira issue. thx. It
should be possible to reuse an existing task for some of the iterations, but
it is important that no wakeup request is ignored so that the the
destination keeps responding.

2009/11/12 afei <afei1689@126.com>

>
> when consuming a large of messages,the method:asyncWakeup() is invoked
> crazily,so the executor
> has a great deal of  runnable that callback Queue.iterate(), but
> Queue.iterate() is much slower than the increasing of runnable in the
> executor. this result in OOM.
> the picture of memory dump:
>
> avoid to invoke the method:asyncWakeup()  frequently???
>
>
> Gary Tully wrote:
> >
> > iterate actually does a dispatch. when consumers change etc, we again
> > try and dispatch to ensure prefetch buffers are maintained.
> >
> > Do a svn up to r834579 as I committed a fix to trunk for this today
> > resolving https://issues.apache.org/activemq/browse/AMQ-2481
> >
> > 2009/11/10 afei <afei1689@126.com>:
> >>
> >> in org.apache.activemq.broker.region ,why so many invoke asyncWakeup(),
> >> what is the method:iterate() doing?
> >>
> >>
> >> afei wrote:
> >>>
> >>> in addition,another problem of OOM.
> >>>
> >>>
> >>>
> >>> Gary Tully wrote:
> >>>>
> >>>> fyi: you can disable periodic message expiry processing using a
> >>>> destination policy entry that sets expireMessagesPeriod = 0
> >>>>
> >>>> 2009/11/9 afei <afei1689@126.com>:
> >>>>>
> >>>>>
> >>>>> when a large of messages in queue,and no consumer or the consumer
is
> >>>>> very
> >>>>> slow, the OOM problem occur, because :
> >>>>> in org.apache.activemq.broker.region.Queue,the 588 line is :
> >>>>>  doBrowse(true, browsedMessages, this.getMaxExpirePageSize());
> >>>>> ,transform to :
> >>>>> doBrowse(false, browsedMessages, this.getMaxExpirePageSize());
> >>>>>  is ok.
> >>>>>
> >>>>>
> >>>>> Dejan Bosanac wrote:
> >>>>>>
> >>>>>> Hi Mitch,
> >>>>>>
> >>>>>> yeah, I said in thread I was referring to, that it is working
with
> >>>>>> "regular"
> >>>>>> stomp connector. I started investigating AMQ-2440 patch the
other
> >>>>>> day,
> >>>>>> should be have something soon.
> >>>>>>
> >>>>>> Cheers
> >>>>>> --
> >>>>>> Dejan Bosanac - http://twitter.com/dejanb
> >>>>>>
> >>>>>> Open Source Integration - http://fusesource.com/
> >>>>>> ActiveMQ in Action - http://www.manning.com/snyder/
> >>>>>> Blog - http://www.nighttale.net
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Oct 28, 2009 at 6:18 PM, Mitch Granger
> >>>>>> <mitch.granger@sophos.com>wrote:
> >>>>>>
> >>>>>>> So we turned off stomp+nio and went back to plain old stomp
and so
> >>>>>>> far
> >>>>>>> it's
> >>>>>>> working fine.  New(IO) isn't always better, I guess :-)
> >>>>>>>
> >>>>>>> Seems like maybe it's this issue ->
> >>>>>>> https://issues.apache.org/activemq/browse/AMQ-2440
> >>>>>>>
> >>>>>>>
> >>>>>>> afei wrote:
> >>>>>>>
> >>>>>>>> i have same problem
> >>>>>>>>
> >>>>>>>> http://www.nabble.com/file/p26093204/aaaaaa.jpg aaaaaa.jpg
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> themitchy wrote:
> >>>>>>>>
> >>>>>>>>> This is what we've done to tune so far:
> >>>>>>>>>
> >>>>>>>>>  - UseDedicatedTaskRunner=false
> >>>>>>>>>  - flow control is off
> >>>>>>>>>  - stomp transport uses transport.closeAsync=false
> >>>>>>>>>
> >>>>>>>>> I agree that it is because of the high number of
open/close
> >>>>>>>>> connections
> >>>>>>>>> from Stomp.  When we monitor through JConsole we
can see more
> >>>>>>>>> threads
> >>>>>>>>> starting up for each new connection.  The problem
is that these
> >>>>>>>>> threads
> >>>>>>>>> don't get let go.  Even though the stomp clients
are
> disconnecting
> >>>>>>>>> the
> >>>>>>>>> number of threads that get released is less than
the number
> >>>>>>>>> created.
> >>>>>>>>> So
> >>>>>>>>>  the thread count goes up and up until it fails.
 All of the
> above
> >>>>>>>>> settings/tuning only delay when it will hit the
wall.
> >>>>>>>>>
> >>>>>>>>> Dejan Bosanac wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Mitch,
> >>>>>>>>>>
> >>>>>>>>>> I think the root cause of this problem is that
you probably have
> >>>>>>>>>> Stomp
> >>>>>>>>>> clients that open/close connection at a high
rate. I simulated
> >>>>>>>>>> this
> >>>>>>>>>> problem
> >>>>>>>>>> on OSX with a StompLoadTest (
> >>>>>>>>>>
> >>>>>>>>>>
> http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/transport/stomp/StompLoadTest.java?view=log
> >>>>>>>>>> ),
> >>>>>>>>>> while trying to reproduce "too many open files"
problem. You can
> >>>>>>>>>> find
> >>>>>>>>>> some
> >>>>>>>>>> of my findings (and workaround) in this thread.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tt25888831.html#a26010080
> >>>>>>>>>>
> >>>>>>>>>> (BTW. it is producing "too many open files"
problem on linux)
> >>>>>>>>>> Basically,
> >>>>>>>>>> the
> >>>>>>>>>> problem with stomp is that every send is done
in separate
> >>>>>>>>>> connection
> >>>>>>>>>> and
> >>>>>>>>>> thus considered to be a new producer for every
message. So when
> >>>>>>>>>> producer
> >>>>>>>>>> flow control is hit, the producers are piling
up and probably
> not
> >>>>>>>>>> releasing
> >>>>>>>>>> connections. Thus you can observe large number
of tcp
> connections
> >>>>>>>>>> on
> >>>>>>>>>> the
> >>>>>>>>>> system in state TIME_WAIT (and TIME_CLOSE),
which causes that
> the
> >>>>>>>>>> system
> >>>>>>>>>> limit is hit at one point. In the above thread,
you can find a
> >>>>>>>>>> workaround
> >>>>>>>>>> that worked for me for that test. I started
investigating this
> >>>>>>>>>> more
> >>>>>>>>>> and
> >>>>>>>>>> hopefully I'll have some more findings in the
near future.
> >>>>>>>>>>
> >>>>>>>>>> Cheers
> >>>>>>>>>> --
> >>>>>>>>>> Dejan Bosanac - http://twitter.com/dejanb
> >>>>>>>>>>
> >>>>>>>>>> Open Source Integration - http://fusesource.com/
> >>>>>>>>>> ActiveMQ in Action - http://www.manning.com/snyder/
> >>>>>>>>>> Blog - http://www.nighttale.net
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Oct 27, 2009 at 1:07 AM, Mitch Granger
> >>>>>>>>>> <mitch.granger@sophos.com>wrote:
> >>>>>>>>>>
> >>>>>>>>>>  Update: We've [nearly] proven that this only
happens with AMQ
> >>>>>>>>>> running
> >>>>>>>>>>> on
> >>>>>>>>>>> openVZ.  What exactly is causing it, we're
still not sure.
> >>>>>>>>>>>  After
> >>>>>>>>>>> memoryUsage is met, the number of threads
skyrockets until we
> >>>>>>>>>>> get
> >>>>>>>>>>> OutOfMemoryError.
> >>>>>>>>>>>
> >>>>>>>>>>> It works just fine on regular hardware;
We're going to try
> >>>>>>>>>>> VMWare
> >>>>>>>>>>> tomorrow.
> >>>>>>>>>>>
> >>>>>>>>>>> One thing really worth mentioning is that
by using the
> >>>>>>>>>>> fileCursor
> >>>>>>>>>>> we
> >>>>>>>>>>> actually started seeing it use the Temp
Store.  When reading
> >>>>>>>>>>> about
> >>>>>>>>>>> systemUsage it is NOT intuitive that the
Temp Store does not
> >>>>>>>>>>> come
> >>>>>>>>>>> into
> >>>>>>>>>>> play
> >>>>>>>>>>> with the default cursor.  Anyone keeping
a significant volume
> of
> >>>>>>>>>>> messages on
> >>>>>>>>>>> their queues should be well served by changing
the cursor.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Mitch Granger wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>  Config is attached.  We have also tried
the
> >>>>>>>>>>> activemq-scalability.xml
> >>>>>>>>>>>> with
> >>>>>>>>>>>> the only change being adding a stomp
connector.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Once we hit the memoryUsage limit we
can [sometimes] connect
> >>>>>>>>>>>> new
> >>>>>>>>>>>> consumers
> >>>>>>>>>>>> but nothing comes back after we send
the SUBSCRIBE frame.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I expect sending to fail when we hit
this limit but if we
> can't
> >>>>>>>>>>>> subscribe
> >>>>>>>>>>>> there's no chance of recovering from
this state.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Rob Davies wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>  On 26 Oct 2009, at 17:38, themitchy
wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>  We're using only persistent messages
and heap size is set to
> >>>>>>>>>>>>> 2GB
> >>>>>>>>>>>>> yet
> >>>>>>>>>>>>> we
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> hit
> >>>>>>>>>>>>>> the memoryUsage limit quite
quickly (system usage config
> >>>>>>>>>>>>>> below).
> >>>>>>>>>>>>>> This
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>> followed by "java.lang.OutOfMemoryError:
unable to create
> new
> >>>>>>>>>>>>>> native
> >>>>>>>>>>>>>>  thread"
> >>>>>>>>>>>>>> as the process quickly reaches
the 2GB of heap we gave it.
> >>>>>>>>>>>>>>  How
> >>>>>>>>>>>>>> are
> >>>>>>>>>>>>>> we
> >>>>>>>>>>>>>> getting to that point with the
memoryUsage limit set far
> >>>>>>>>>>>>>> below
> >>>>>>>>>>>>>> it?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Is there no way to get AMQ to
gracefully limit it's memory
> >>>>>>>>>>>>>> usage?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>      <systemUsage>
> >>>>>>>>>>>>>>          <systemUsage>
> >>>>>>>>>>>>>>              <memoryUsage>
> >>>>>>>>>>>>>>                  <memoryUsage
limit="256 mb"/>
> >>>>>>>>>>>>>>              </memoryUsage>
> >>>>>>>>>>>>>>              <storeUsage>
> >>>>>>>>>>>>>>                  <storeUsage
limit="60 gb" name="foo"/>
> >>>>>>>>>>>>>>              </storeUsage>
> >>>>>>>>>>>>>>              <tempUsage>
> >>>>>>>>>>>>>>                  <tempUsage
limit="60 gb"/>
> >>>>>>>>>>>>>>              </tempUsage>
> >>>>>>>>>>>>>>          </systemUsage>
> >>>>>>>>>>>>>>      </systemUsage>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>> View this message in context:
> >>>>>>>>>>>>>>
> http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26064098.html
> >>>>>>>>>>>>>> Sent from the ActiveMQ - User
mailing list archive at
> >>>>>>>>>>>>>> Nabble.com.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>  Can you send the rest of your
config ?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Rob Davies
> >>>>>>>>>>>>> http://twitter.com/rajdavies
> >>>>>>>>>>>>> I work here: http://fusesource.com
> >>>>>>>>>>>>> My Blog: http://rajdavies.blogspot.com/
> >>>>>>>>>>>>> I'm writing this: http://www.manning.com/snyder/
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> -----
> >>>>>> Dejan Bosanac
> >>>>>>
> >>>>>> Open Source Integration - http://fusesource.com/
> >>>>>> ActiveMQ in Action - http://www.manning.com/snyder/
> >>>>>> Blog - http://www.nighttale.net
> >>>>>>
> >>>>> http://old.nabble.com/file/p26264779/dump.jpg
> >>>>> --
> >>>>> View this message in context:
> >>>>> http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26264779.html
> >>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> http://blog.garytully.com
> >>>>
> >>>> Open Source Integration
> >>>> http://fusesource.com
> >>>>
> >>>>
> >>>  http://old.nabble.com/file/p26278228/oom.jpg
> >>>
> >>
> >> --
> >> View this message in context:
> >> http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26279671.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> >
> > --
> > http://blog.garytully.com
> >
> > Open Source Integration
> > http://fusesource.com
> >
> >
> http://old.nabble.com/file/p26312669/oom.jpg
> --
> View this message in context:
> http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26312669.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>


-- 
http://blog.garytully.com

Open Source Integration
http://fusesource.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message