activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <gary.tu...@gmail.com>
Subject Re: Out of Memory on 5.3
Date Wed, 18 Nov 2009 09:39:15 GMT
in the next few weeks, we don't yet have a hard date.

2009/11/18 afei <afei1689@126.com>

>
> Gary Tully,thanks , when is activemq5.3.1 released?
> we urgently use it.
>
>
>
> Gary Tully wrote:
> >
> > a similar fix, but removing the deterministic task runner and reverting
> to
> > the pooled/dedicated task runners with a wake-up count does the trick.
> The
> > queue size in this case is always < 2.
> > Can you do an svn up to validate.
> >
> > thanks for the heads up on your fix. Think it is best not to limit the
> > pending wakeups to max page size as it may result in a hung destination.
> >
> > 2009/11/13 afei <afei1689@126.com>
> >
> >>
> >> the test scene: after accumulate a large of messages(about 5 millions)
> in
> >> queue,begin to consume it.
> >>
> >> i do a modifications,look at the attach (
> >> http://old.nabble.com/file/p26328726/Queue.java Queue.java ),it look
> like
> >> ok.
> >> the test xml configuration.
> >>
> >>
> >> Gary Tully wrote:
> >> >
> >> > https://issues.apache.org/activemq/browse/AMQ-2483 is tracking
> >> this.Could
> >> > you attach your test case and/or configuration to the jira issue. thx.
> >> It
> >> > should be possible to reuse an existing task for some of the
> >> iterations,
> >> > but
> >> > it is important that no wakeup request is ignored so that the the
> >> > destination keeps responding.
> >> >
> >> > 2009/11/12 afei <afei1689@126.com>
> >> >
> >> >>
> >> >> when consuming a large of messages,the method:asyncWakeup() is
> invoked
> >> >> crazily,so the executor
> >> >> has a great deal of  runnable that callback Queue.iterate(), but
> >> >> Queue.iterate() is much slower than the increasing of runnable in the
> >> >> executor. this result in OOM.
> >> >> the picture of memory dump:
> >> >>
> >> >> avoid to invoke the method:asyncWakeup()  frequently???
> >> >>
> >> >>
> >> >> Gary Tully wrote:
> >> >> >
> >> >> > iterate actually does a dispatch. when consumers change etc, we
> >> again
> >> >> > try and dispatch to ensure prefetch buffers are maintained.
> >> >> >
> >> >> > Do a svn up to r834579 as I committed a fix to trunk for this
today
> >> >> > resolving https://issues.apache.org/activemq/browse/AMQ-2481
> >> >> >
> >> >> > 2009/11/10 afei <afei1689@126.com>:
> >> >> >>
> >> >> >> in org.apache.activemq.broker.region ,why so many invoke
> >> >> asyncWakeup(),
> >> >> >> what is the method:iterate() doing?
> >> >> >>
> >> >> >>
> >> >> >> afei wrote:
> >> >> >>>
> >> >> >>> in addition,another problem of OOM.
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>> Gary Tully wrote:
> >> >> >>>>
> >> >> >>>> fyi: you can disable periodic message expiry processing
using a
> >> >> >>>> destination policy entry that sets expireMessagesPeriod
= 0
> >> >> >>>>
> >> >> >>>> 2009/11/9 afei <afei1689@126.com>:
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> when a large of messages in queue,and no consumer
or the
> >> consumer
> >> >> is
> >> >> >>>>> very
> >> >> >>>>> slow, the OOM problem occur, because :
> >> >> >>>>> in org.apache.activemq.broker.region.Queue,the
588 line is :
> >> >> >>>>>  doBrowse(true, browsedMessages, this.getMaxExpirePageSize());
> >> >> >>>>> ,transform to :
> >> >> >>>>> doBrowse(false, browsedMessages, this.getMaxExpirePageSize());
> >> >> >>>>>  is ok.
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> Dejan Bosanac wrote:
> >> >> >>>>>>
> >> >> >>>>>> Hi Mitch,
> >> >> >>>>>>
> >> >> >>>>>> yeah, I said in thread I was referring to,
that it is working
> >> with
> >> >> >>>>>> "regular"
> >> >> >>>>>> stomp connector. I started investigating AMQ-2440
patch the
> >> other
> >> >> >>>>>> day,
> >> >> >>>>>> should be have something soon.
> >> >> >>>>>>
> >> >> >>>>>> Cheers
> >> >> >>>>>> --
> >> >> >>>>>> Dejan Bosanac - http://twitter.com/dejanb
> >> >> >>>>>>
> >> >> >>>>>> Open Source Integration - http://fusesource.com/
> >> >> >>>>>> ActiveMQ in Action - http://www.manning.com/snyder/
> >> >> >>>>>> Blog - http://www.nighttale.net
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>> On Wed, Oct 28, 2009 at 6:18 PM, Mitch Granger
> >> >> >>>>>> <mitch.granger@sophos.com>wrote:
> >> >> >>>>>>
> >> >> >>>>>>> So we turned off stomp+nio and went back
to plain old stomp
> >> and
> >> >> so
> >> >> >>>>>>> far
> >> >> >>>>>>> it's
> >> >> >>>>>>> working fine.  New(IO) isn't always better,
I guess :-)
> >> >> >>>>>>>
> >> >> >>>>>>> Seems like maybe it's this issue ->
> >> >> >>>>>>> https://issues.apache.org/activemq/browse/AMQ-2440
> >> >> >>>>>>>
> >> >> >>>>>>>
> >> >> >>>>>>> afei wrote:
> >> >> >>>>>>>
> >> >> >>>>>>>> i have same problem
> >> >> >>>>>>>>
> >> >> >>>>>>>> http://www.nabble.com/file/p26093204/aaaaaa.jpg
aaaaaa.jpg
> >> >> >>>>>>>>
> >> >> >>>>>>>>
> >> >> >>>>>>>> themitchy wrote:
> >> >> >>>>>>>>
> >> >> >>>>>>>>> This is what we've done to tune
so far:
> >> >> >>>>>>>>>
> >> >> >>>>>>>>>  - UseDedicatedTaskRunner=false
> >> >> >>>>>>>>>  - flow control is off
> >> >> >>>>>>>>>  - stomp transport uses transport.closeAsync=false
> >> >> >>>>>>>>>
> >> >> >>>>>>>>> I agree that it is because of
the high number of open/close
> >> >> >>>>>>>>> connections
> >> >> >>>>>>>>> from Stomp.  When we monitor through
JConsole we can see
> >> more
> >> >> >>>>>>>>> threads
> >> >> >>>>>>>>> starting up for each new connection.
 The problem is that
> >> these
> >> >> >>>>>>>>> threads
> >> >> >>>>>>>>> don't get let go.  Even though
the stomp clients are
> >> >> disconnecting
> >> >> >>>>>>>>> the
> >> >> >>>>>>>>> number of threads that get released
is less than the number
> >> >> >>>>>>>>> created.
> >> >> >>>>>>>>> So
> >> >> >>>>>>>>>  the thread count goes up and
up until it fails.  All of
> the
> >> >> above
> >> >> >>>>>>>>> settings/tuning only delay when
it will hit the wall.
> >> >> >>>>>>>>>
> >> >> >>>>>>>>> Dejan Bosanac wrote:
> >> >> >>>>>>>>>
> >> >> >>>>>>>>>> Hi Mitch,
> >> >> >>>>>>>>>>
> >> >> >>>>>>>>>> I think the root cause of
this problem is that you
> probably
> >> >> have
> >> >> >>>>>>>>>> Stomp
> >> >> >>>>>>>>>> clients that open/close connection
at a high rate. I
> >> simulated
> >> >> >>>>>>>>>> this
> >> >> >>>>>>>>>> problem
> >> >> >>>>>>>>>> on OSX with a StompLoadTest
(
> >> >> >>>>>>>>>>
> >> >> >>>>>>>>>>
> >> >>
> >>
> http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/transport/stomp/StompLoadTest.java?view=log
> >> >> >>>>>>>>>> ),
> >> >> >>>>>>>>>> while trying to reproduce
"too many open files" problem.
> >> You
> >> >> can
> >> >> >>>>>>>>>> find
> >> >> >>>>>>>>>> some
> >> >> >>>>>>>>>> of my findings (and workaround)
in this thread.
> >> >> >>>>>>>>>>
> >> >> >>>>>>>>>>
> >> >> >>>>>>>>>>
> >> >>
> >>
> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tt25888831.html#a26010080
> >> >> >>>>>>>>>>
> >> >> >>>>>>>>>> (BTW. it is producing "too
many open files" problem on
> >> linux)
> >> >> >>>>>>>>>> Basically,
> >> >> >>>>>>>>>> the
> >> >> >>>>>>>>>> problem with stomp is that
every send is done in separate
> >> >> >>>>>>>>>> connection
> >> >> >>>>>>>>>> and
> >> >> >>>>>>>>>> thus considered to be a new
producer for every message. So
> >> >> when
> >> >> >>>>>>>>>> producer
> >> >> >>>>>>>>>> flow control is hit, the producers
are piling up and
> >> probably
> >> >> not
> >> >> >>>>>>>>>> releasing
> >> >> >>>>>>>>>> connections. Thus you can
observe large number of tcp
> >> >> connections
> >> >> >>>>>>>>>> on
> >> >> >>>>>>>>>> the
> >> >> >>>>>>>>>> system in state TIME_WAIT
(and TIME_CLOSE), which causes
> >> that
> >> >> the
> >> >> >>>>>>>>>> system
> >> >> >>>>>>>>>> limit is hit at one point.
In the above thread, you can
> >> find
> >> a
> >> >> >>>>>>>>>> workaround
> >> >> >>>>>>>>>> that worked for me for that
test. I started investigating
> >> this
> >> >> >>>>>>>>>> more
> >> >> >>>>>>>>>> and
> >> >> >>>>>>>>>> hopefully I'll have some more
findings in the near future.
> >> >> >>>>>>>>>>
> >> >> >>>>>>>>>> Cheers
> >> >> >>>>>>>>>> --
> >> >> >>>>>>>>>> Dejan Bosanac - http://twitter.com/dejanb
> >> >> >>>>>>>>>>
> >> >> >>>>>>>>>> Open Source Integration -
http://fusesource.com/
> >> >> >>>>>>>>>> ActiveMQ in Action - http://www.manning.com/snyder/
> >> >> >>>>>>>>>> Blog - http://www.nighttale.net
> >> >> >>>>>>>>>>
> >> >> >>>>>>>>>>
> >> >> >>>>>>>>>> On Tue, Oct 27, 2009 at 1:07
AM, Mitch Granger
> >> >> >>>>>>>>>> <mitch.granger@sophos.com>wrote:
> >> >> >>>>>>>>>>
> >> >> >>>>>>>>>>  Update: We've [nearly] proven
that this only happens with
> >> AMQ
> >> >> >>>>>>>>>> running
> >> >> >>>>>>>>>>> on
> >> >> >>>>>>>>>>> openVZ.  What exactly
is causing it, we're still not
> sure.
> >> >> >>>>>>>>>>>  After
> >> >> >>>>>>>>>>> memoryUsage is met, the
number of threads skyrockets
> until
> >> we
> >> >> >>>>>>>>>>> get
> >> >> >>>>>>>>>>> OutOfMemoryError.
> >> >> >>>>>>>>>>>
> >> >> >>>>>>>>>>> It works just fine on
regular hardware; We're going to
> try
> >> >> >>>>>>>>>>> VMWare
> >> >> >>>>>>>>>>> tomorrow.
> >> >> >>>>>>>>>>>
> >> >> >>>>>>>>>>> One thing really worth
mentioning is that by using the
> >> >> >>>>>>>>>>> fileCursor
> >> >> >>>>>>>>>>> we
> >> >> >>>>>>>>>>> actually started seeing
it use the Temp Store.  When
> >> reading
> >> >> >>>>>>>>>>> about
> >> >> >>>>>>>>>>> systemUsage it is NOT
intuitive that the Temp Store does
> >> not
> >> >> >>>>>>>>>>> come
> >> >> >>>>>>>>>>> into
> >> >> >>>>>>>>>>> play
> >> >> >>>>>>>>>>> with the default cursor.
 Anyone keeping a significant
> >> volume
> >> >> of
> >> >> >>>>>>>>>>> messages on
> >> >> >>>>>>>>>>> their queues should be
well served by changing the
> cursor.
> >> >> >>>>>>>>>>>
> >> >> >>>>>>>>>>>
> >> >> >>>>>>>>>>> Mitch Granger wrote:
> >> >> >>>>>>>>>>>
> >> >> >>>>>>>>>>>  Config is attached. 
We have also tried the
> >> >> >>>>>>>>>>> activemq-scalability.xml
> >> >> >>>>>>>>>>>> with
> >> >> >>>>>>>>>>>> the only change being
adding a stomp connector.
> >> >> >>>>>>>>>>>>
> >> >> >>>>>>>>>>>> Once we hit the memoryUsage
limit we can [sometimes]
> >> connect
> >> >> >>>>>>>>>>>> new
> >> >> >>>>>>>>>>>> consumers
> >> >> >>>>>>>>>>>> but nothing comes
back after we send the SUBSCRIBE
> frame.
> >> >> >>>>>>>>>>>>
> >> >> >>>>>>>>>>>> I expect sending to
fail when we hit this limit but if
> we
> >> >> can't
> >> >> >>>>>>>>>>>> subscribe
> >> >> >>>>>>>>>>>> there's no chance
of recovering from this state.
> >> >> >>>>>>>>>>>>
> >> >> >>>>>>>>>>>> Rob Davies wrote:
> >> >> >>>>>>>>>>>>
> >> >> >>>>>>>>>>>>  On 26 Oct 2009, at
17:38, themitchy wrote:
> >> >> >>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>  We're using only
persistent messages and heap size is
> >> set
> >> >> to
> >> >> >>>>>>>>>>>>> 2GB
> >> >> >>>>>>>>>>>>> yet
> >> >> >>>>>>>>>>>>> we
> >> >> >>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>> hit
> >> >> >>>>>>>>>>>>>> the memoryUsage
limit quite quickly (system usage
> >> config
> >> >> >>>>>>>>>>>>>> below).
> >> >> >>>>>>>>>>>>>> This
> >> >> >>>>>>>>>>>>>> is
> >> >> >>>>>>>>>>>>>> followed by
"java.lang.OutOfMemoryError: unable to
> >> create
> >> >> new
> >> >> >>>>>>>>>>>>>> native
> >> >> >>>>>>>>>>>>>>  thread"
> >> >> >>>>>>>>>>>>>> as the process
quickly reaches the 2GB of heap we gave
> >> it.
> >> >> >>>>>>>>>>>>>>  How
> >> >> >>>>>>>>>>>>>> are
> >> >> >>>>>>>>>>>>>> we
> >> >> >>>>>>>>>>>>>> getting to
that point with the memoryUsage limit set
> >> far
> >> >> >>>>>>>>>>>>>> below
> >> >> >>>>>>>>>>>>>> it?
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>> Is there no
way to get AMQ to gracefully limit it's
> >> memory
> >> >> >>>>>>>>>>>>>> usage?
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>>      <systemUsage>
> >> >> >>>>>>>>>>>>>>          <systemUsage>
> >> >> >>>>>>>>>>>>>>          
   <memoryUsage>
> >> >> >>>>>>>>>>>>>>          
       <memoryUsage limit="256 mb"/>
> >> >> >>>>>>>>>>>>>>          
   </memoryUsage>
> >> >> >>>>>>>>>>>>>>          
   <storeUsage>
> >> >> >>>>>>>>>>>>>>          
       <storeUsage limit="60 gb"
> name="foo"/>
> >> >> >>>>>>>>>>>>>>          
   </storeUsage>
> >> >> >>>>>>>>>>>>>>          
   <tempUsage>
> >> >> >>>>>>>>>>>>>>          
       <tempUsage limit="60 gb"/>
> >> >> >>>>>>>>>>>>>>          
   </tempUsage>
> >> >> >>>>>>>>>>>>>>          </systemUsage>
> >> >> >>>>>>>>>>>>>>      </systemUsage>
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>> --
> >> >> >>>>>>>>>>>>>> View this
message in context:
> >> >> >>>>>>>>>>>>>>
> >> >> http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26064098.html
> >> >> >>>>>>>>>>>>>> Sent from
the ActiveMQ - User mailing list archive at
> >> >> >>>>>>>>>>>>>> Nabble.com.
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>>  Can you send
the rest of your config ?
> >> >> >>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>> Rob Davies
> >> >> >>>>>>>>>>>>> http://twitter.com/rajdavies
> >> >> >>>>>>>>>>>>> I work here: http://fusesource.com
> >> >> >>>>>>>>>>>>> My Blog: http://rajdavies.blogspot.com/
> >> >> >>>>>>>>>>>>> I'm writing this:
http://www.manning.com/snyder/
> >> >> >>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>
> >> >> >>>>>>>>>
> >> >> >>>>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>> -----
> >> >> >>>>>> Dejan Bosanac
> >> >> >>>>>>
> >> >> >>>>>> Open Source Integration - http://fusesource.com/
> >> >> >>>>>> ActiveMQ in Action - http://www.manning.com/snyder/
> >> >> >>>>>> Blog - http://www.nighttale.net
> >> >> >>>>>>
> >> >> >>>>> http://old.nabble.com/file/p26264779/dump.jpg
> >> >> >>>>> --
> >> >> >>>>> View this message in context:
> >> >> >>>>>
> >> http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26264779.html
> >> >> >>>>> Sent from the ActiveMQ - User mailing list archive
at
> >> Nabble.com.
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>
> >> >> >>>>
> >> >> >>>>
> >> >> >>>> --
> >> >> >>>> http://blog.garytully.com
> >> >> >>>>
> >> >> >>>> Open Source Integration
> >> >> >>>> http://fusesource.com
> >> >> >>>>
> >> >> >>>>
> >> >> >>>  http://old.nabble.com/file/p26278228/oom.jpg
> >> >> >>>
> >> >> >>
> >> >> >> --
> >> >> >> View this message in context:
> >> >> >>
> http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26279671.html
> >> >> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >> >> >>
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > http://blog.garytully.com
> >> >> >
> >> >> > Open Source Integration
> >> >> > http://fusesource.com
> >> >> >
> >> >> >
> >> >> http://old.nabble.com/file/p26312669/oom.jpg
> >> >> --
> >> >> View this message in context:
> >> >> http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26312669.html
> >> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >> >>
> >> >>
> >> >
> >> >
> >> > --
> >> > http://blog.garytully.com
> >> >
> >> > Open Source Integration
> >> > http://fusesource.com
> >> >
> >> >
> >> http://old.nabble.com/file/p26328726/activemq.xml activemq.xml
> >> http://old.nabble.com/file/p26328726/activemq.xml activemq.xml
> >> --
> >> View this message in context:
> >> http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26328726.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > http://blog.garytully.com
> >
> > Open Source Integration
> > http://fusesource.com
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26401884.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>


-- 
http://blog.garytully.com

Open Source Integration
http://fusesource.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message