activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <gary.tu...@gmail.com>
Subject Re: Out of Memory on 5.3
Date Tue, 10 Nov 2009 18:24:27 GMT
iterate actually does a dispatch. when consumers change etc, we again
try and dispatch to ensure prefetch buffers are maintained.

Do a svn up to r834579 as I committed a fix to trunk for this today
resolving https://issues.apache.org/activemq/browse/AMQ-2481

2009/11/10 afei <afei1689@126.com>:
>
> in org.apache.activemq.broker.region ,why so many invoke asyncWakeup(),
> what is the method:iterate() doing?
>
>
> afei wrote:
>>
>> in addition,another problem of OOM.
>>
>>
>>
>> Gary Tully wrote:
>>>
>>> fyi: you can disable periodic message expiry processing using a
>>> destination policy entry that sets expireMessagesPeriod = 0
>>>
>>> 2009/11/9 afei <afei1689@126.com>:
>>>>
>>>>
>>>> when a large of messages in queue,and no consumer or the consumer is
>>>> very
>>>> slow, the OOM problem occur, because :
>>>> in org.apache.activemq.broker.region.Queue,the 588 line is :
>>>>  doBrowse(true, browsedMessages, this.getMaxExpirePageSize());
>>>> ,transform to :
>>>> doBrowse(false, browsedMessages, this.getMaxExpirePageSize());
>>>>  is ok.
>>>>
>>>>
>>>> Dejan Bosanac wrote:
>>>>>
>>>>> Hi Mitch,
>>>>>
>>>>> yeah, I said in thread I was referring to, that it is working with
>>>>> "regular"
>>>>> stomp connector. I started investigating AMQ-2440 patch the other day,
>>>>> should be have something soon.
>>>>>
>>>>> Cheers
>>>>> --
>>>>> Dejan Bosanac - http://twitter.com/dejanb
>>>>>
>>>>> Open Source Integration - http://fusesource.com/
>>>>> ActiveMQ in Action - http://www.manning.com/snyder/
>>>>> Blog - http://www.nighttale.net
>>>>>
>>>>>
>>>>> On Wed, Oct 28, 2009 at 6:18 PM, Mitch Granger
>>>>> <mitch.granger@sophos.com>wrote:
>>>>>
>>>>>> So we turned off stomp+nio and went back to plain old stomp and so
far
>>>>>> it's
>>>>>> working fine.  New(IO) isn't always better, I guess :-)
>>>>>>
>>>>>> Seems like maybe it's this issue ->
>>>>>> https://issues.apache.org/activemq/browse/AMQ-2440
>>>>>>
>>>>>>
>>>>>> afei wrote:
>>>>>>
>>>>>>> i have same problem
>>>>>>>
>>>>>>> http://www.nabble.com/file/p26093204/aaaaaa.jpg aaaaaa.jpg
>>>>>>>
>>>>>>>
>>>>>>> themitchy wrote:
>>>>>>>
>>>>>>>> This is what we've done to tune so far:
>>>>>>>>
>>>>>>>>  - UseDedicatedTaskRunner=false
>>>>>>>>  - flow control is off
>>>>>>>>  - stomp transport uses transport.closeAsync=false
>>>>>>>>
>>>>>>>> I agree that it is because of the high number of open/close
>>>>>>>> connections
>>>>>>>> from Stomp.  When we monitor through JConsole we can see
more
>>>>>>>> threads
>>>>>>>> starting up for each new connection.  The problem is that
these
>>>>>>>> threads
>>>>>>>> don't get let go.  Even though the stomp clients are disconnecting
>>>>>>>> the
>>>>>>>> number of threads that get released is less than the number
created.
>>>>>>>> So
>>>>>>>>  the thread count goes up and up until it fails.  All of
the above
>>>>>>>> settings/tuning only delay when it will hit the wall.
>>>>>>>>
>>>>>>>> Dejan Bosanac wrote:
>>>>>>>>
>>>>>>>>> Hi Mitch,
>>>>>>>>>
>>>>>>>>> I think the root cause of this problem is that you probably
have
>>>>>>>>> Stomp
>>>>>>>>> clients that open/close connection at a high rate. I
simulated this
>>>>>>>>> problem
>>>>>>>>> on OSX with a StompLoadTest (
>>>>>>>>>
>>>>>>>>> http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/transport/stomp/StompLoadTest.java?view=log
>>>>>>>>> ),
>>>>>>>>> while trying to reproduce "too many open files" problem.
You can
>>>>>>>>> find
>>>>>>>>> some
>>>>>>>>> of my findings (and workaround) in this thread.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tt25888831.html#a26010080
>>>>>>>>>
>>>>>>>>> (BTW. it is producing "too many open files" problem on
linux)
>>>>>>>>> Basically,
>>>>>>>>> the
>>>>>>>>> problem with stomp is that every send is done in separate
>>>>>>>>> connection
>>>>>>>>> and
>>>>>>>>> thus considered to be a new producer for every message.
So when
>>>>>>>>> producer
>>>>>>>>> flow control is hit, the producers are piling up and
probably not
>>>>>>>>> releasing
>>>>>>>>> connections. Thus you can observe large number of tcp
connections
>>>>>>>>> on
>>>>>>>>> the
>>>>>>>>> system in state TIME_WAIT (and TIME_CLOSE), which causes
that the
>>>>>>>>> system
>>>>>>>>> limit is hit at one point. In the above thread, you can
find a
>>>>>>>>> workaround
>>>>>>>>> that worked for me for that test. I started investigating
this more
>>>>>>>>> and
>>>>>>>>> hopefully I'll have some more findings in the near future.
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>> --
>>>>>>>>> Dejan Bosanac - http://twitter.com/dejanb
>>>>>>>>>
>>>>>>>>> Open Source Integration - http://fusesource.com/
>>>>>>>>> ActiveMQ in Action - http://www.manning.com/snyder/
>>>>>>>>> Blog - http://www.nighttale.net
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Oct 27, 2009 at 1:07 AM, Mitch Granger
>>>>>>>>> <mitch.granger@sophos.com>wrote:
>>>>>>>>>
>>>>>>>>>  Update: We've [nearly] proven that this only happens
with AMQ
>>>>>>>>> running
>>>>>>>>>> on
>>>>>>>>>> openVZ.  What exactly is causing it, we're still
not sure.  After
>>>>>>>>>> memoryUsage is met, the number of threads skyrockets
until we get
>>>>>>>>>> OutOfMemoryError.
>>>>>>>>>>
>>>>>>>>>> It works just fine on regular hardware; We're going
to try VMWare
>>>>>>>>>> tomorrow.
>>>>>>>>>>
>>>>>>>>>> One thing really worth mentioning is that by using
the fileCursor
>>>>>>>>>> we
>>>>>>>>>> actually started seeing it use the Temp Store.  When
reading about
>>>>>>>>>> systemUsage it is NOT intuitive that the Temp Store
does not come
>>>>>>>>>> into
>>>>>>>>>> play
>>>>>>>>>> with the default cursor.  Anyone keeping a significant
volume of
>>>>>>>>>> messages on
>>>>>>>>>> their queues should be well served by changing the
cursor.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Mitch Granger wrote:
>>>>>>>>>>
>>>>>>>>>>  Config is attached.  We have also tried the
>>>>>>>>>> activemq-scalability.xml
>>>>>>>>>>> with
>>>>>>>>>>> the only change being adding a stomp connector.
>>>>>>>>>>>
>>>>>>>>>>> Once we hit the memoryUsage limit we can [sometimes]
connect new
>>>>>>>>>>> consumers
>>>>>>>>>>> but nothing comes back after we send the SUBSCRIBE
frame.
>>>>>>>>>>>
>>>>>>>>>>> I expect sending to fail when we hit this limit
but if we can't
>>>>>>>>>>> subscribe
>>>>>>>>>>> there's no chance of recovering from this state.
>>>>>>>>>>>
>>>>>>>>>>> Rob Davies wrote:
>>>>>>>>>>>
>>>>>>>>>>>  On 26 Oct 2009, at 17:38, themitchy wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>  We're using only persistent messages and
heap size is set to
>>>>>>>>>>>> 2GB
>>>>>>>>>>>> yet
>>>>>>>>>>>> we
>>>>>>>>>>>>
>>>>>>>>>>>>> hit
>>>>>>>>>>>>> the memoryUsage limit quite quickly (system
usage config
>>>>>>>>>>>>> below).
>>>>>>>>>>>>> This
>>>>>>>>>>>>> is
>>>>>>>>>>>>> followed by "java.lang.OutOfMemoryError:
unable to create new
>>>>>>>>>>>>> native
>>>>>>>>>>>>>  thread"
>>>>>>>>>>>>> as the process quickly reaches the 2GB
of heap we gave it.  How
>>>>>>>>>>>>> are
>>>>>>>>>>>>> we
>>>>>>>>>>>>> getting to that point with the memoryUsage
limit set far below
>>>>>>>>>>>>> it?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is there no way to get AMQ to gracefully
limit it's memory
>>>>>>>>>>>>> usage?
>>>>>>>>>>>>>
>>>>>>>>>>>>>      <systemUsage>
>>>>>>>>>>>>>          <systemUsage>
>>>>>>>>>>>>>              <memoryUsage>
>>>>>>>>>>>>>                  <memoryUsage
limit="256 mb"/>
>>>>>>>>>>>>>              </memoryUsage>
>>>>>>>>>>>>>              <storeUsage>
>>>>>>>>>>>>>                  <storeUsage
limit="60 gb" name="foo"/>
>>>>>>>>>>>>>              </storeUsage>
>>>>>>>>>>>>>              <tempUsage>
>>>>>>>>>>>>>                  <tempUsage
limit="60 gb"/>
>>>>>>>>>>>>>              </tempUsage>
>>>>>>>>>>>>>          </systemUsage>
>>>>>>>>>>>>>      </systemUsage>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> View this message in context:
>>>>>>>>>>>>> http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26064098.html
>>>>>>>>>>>>> Sent from the ActiveMQ - User mailing
list archive at
>>>>>>>>>>>>> Nabble.com.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>  Can you send the rest of your config
?
>>>>>>>>>>>>
>>>>>>>>>>>> Rob Davies
>>>>>>>>>>>> http://twitter.com/rajdavies
>>>>>>>>>>>> I work here: http://fusesource.com
>>>>>>>>>>>> My Blog: http://rajdavies.blogspot.com/
>>>>>>>>>>>> I'm writing this: http://www.manning.com/snyder/
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>> -----
>>>>> Dejan Bosanac
>>>>>
>>>>> Open Source Integration - http://fusesource.com/
>>>>> ActiveMQ in Action - http://www.manning.com/snyder/
>>>>> Blog - http://www.nighttale.net
>>>>>
>>>> http://old.nabble.com/file/p26264779/dump.jpg
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26264779.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> http://blog.garytully.com
>>>
>>> Open Source Integration
>>> http://fusesource.com
>>>
>>>
>>  http://old.nabble.com/file/p26278228/oom.jpg
>>
>
> --
> View this message in context: http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26279671.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>



-- 
http://blog.garytully.com

Open Source Integration
http://fusesource.com

Mime
View raw message