activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From afei <afei1...@126.com>
Subject Re: Out of Memory on 5.3
Date Tue, 10 Nov 2009 08:06:18 GMT

in org.apache.activemq.broker.region ,why so many invoke asyncWakeup(),
what is the method:iterate() doing?


afei wrote:
> 
> in addition,another problem of OOM.
> 
> 
> 
> Gary Tully wrote:
>> 
>> fyi: you can disable periodic message expiry processing using a
>> destination policy entry that sets expireMessagesPeriod = 0
>> 
>> 2009/11/9 afei <afei1689@126.com>:
>>>
>>>
>>> when a large of messages in queue,and no consumer or the consumer is
>>> very
>>> slow, the OOM problem occur, because :
>>> in org.apache.activemq.broker.region.Queue,the 588 line is :
>>>  doBrowse(true, browsedMessages, this.getMaxExpirePageSize());
>>> ,transform to :
>>> doBrowse(false, browsedMessages, this.getMaxExpirePageSize());
>>>  is ok.
>>>
>>>
>>> Dejan Bosanac wrote:
>>>>
>>>> Hi Mitch,
>>>>
>>>> yeah, I said in thread I was referring to, that it is working with
>>>> "regular"
>>>> stomp connector. I started investigating AMQ-2440 patch the other day,
>>>> should be have something soon.
>>>>
>>>> Cheers
>>>> --
>>>> Dejan Bosanac - http://twitter.com/dejanb
>>>>
>>>> Open Source Integration - http://fusesource.com/
>>>> ActiveMQ in Action - http://www.manning.com/snyder/
>>>> Blog - http://www.nighttale.net
>>>>
>>>>
>>>> On Wed, Oct 28, 2009 at 6:18 PM, Mitch Granger
>>>> <mitch.granger@sophos.com>wrote:
>>>>
>>>>> So we turned off stomp+nio and went back to plain old stomp and so far
>>>>> it's
>>>>> working fine.  New(IO) isn't always better, I guess :-)
>>>>>
>>>>> Seems like maybe it's this issue ->
>>>>> https://issues.apache.org/activemq/browse/AMQ-2440
>>>>>
>>>>>
>>>>> afei wrote:
>>>>>
>>>>>> i have same problem
>>>>>>
>>>>>> http://www.nabble.com/file/p26093204/aaaaaa.jpg aaaaaa.jpg
>>>>>>
>>>>>>
>>>>>> themitchy wrote:
>>>>>>
>>>>>>> This is what we've done to tune so far:
>>>>>>>
>>>>>>>  - UseDedicatedTaskRunner=false
>>>>>>>  - flow control is off
>>>>>>>  - stomp transport uses transport.closeAsync=false
>>>>>>>
>>>>>>> I agree that it is because of the high number of open/close
>>>>>>> connections
>>>>>>> from Stomp.  When we monitor through JConsole we can see more
>>>>>>> threads
>>>>>>> starting up for each new connection.  The problem is that these
>>>>>>> threads
>>>>>>> don't get let go.  Even though the stomp clients are disconnecting
>>>>>>> the
>>>>>>> number of threads that get released is less than the number created.
>>>>>>> So
>>>>>>>  the thread count goes up and up until it fails.  All of the
above
>>>>>>> settings/tuning only delay when it will hit the wall.
>>>>>>>
>>>>>>> Dejan Bosanac wrote:
>>>>>>>
>>>>>>>> Hi Mitch,
>>>>>>>>
>>>>>>>> I think the root cause of this problem is that you probably
have
>>>>>>>> Stomp
>>>>>>>> clients that open/close connection at a high rate. I simulated
this
>>>>>>>> problem
>>>>>>>> on OSX with a StompLoadTest (
>>>>>>>>
>>>>>>>> http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/transport/stomp/StompLoadTest.java?view=log
>>>>>>>> ),
>>>>>>>> while trying to reproduce "too many open files" problem.
You can
>>>>>>>> find
>>>>>>>> some
>>>>>>>> of my findings (and workaround) in this thread.
>>>>>>>>
>>>>>>>>
>>>>>>>> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tt25888831.html#a26010080
>>>>>>>>
>>>>>>>> (BTW. it is producing "too many open files" problem on linux)
>>>>>>>> Basically,
>>>>>>>> the
>>>>>>>> problem with stomp is that every send is done in separate
>>>>>>>> connection
>>>>>>>> and
>>>>>>>> thus considered to be a new producer for every message. So
when
>>>>>>>> producer
>>>>>>>> flow control is hit, the producers are piling up and probably
not
>>>>>>>> releasing
>>>>>>>> connections. Thus you can observe large number of tcp connections
>>>>>>>> on
>>>>>>>> the
>>>>>>>> system in state TIME_WAIT (and TIME_CLOSE), which causes
that the
>>>>>>>> system
>>>>>>>> limit is hit at one point. In the above thread, you can find
a
>>>>>>>> workaround
>>>>>>>> that worked for me for that test. I started investigating
this more
>>>>>>>> and
>>>>>>>> hopefully I'll have some more findings in the near future.
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>> --
>>>>>>>> Dejan Bosanac - http://twitter.com/dejanb
>>>>>>>>
>>>>>>>> Open Source Integration - http://fusesource.com/
>>>>>>>> ActiveMQ in Action - http://www.manning.com/snyder/
>>>>>>>> Blog - http://www.nighttale.net
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Oct 27, 2009 at 1:07 AM, Mitch Granger
>>>>>>>> <mitch.granger@sophos.com>wrote:
>>>>>>>>
>>>>>>>>  Update: We've [nearly] proven that this only happens with
AMQ
>>>>>>>> running
>>>>>>>>> on
>>>>>>>>> openVZ.  What exactly is causing it, we're still not
sure.  After
>>>>>>>>> memoryUsage is met, the number of threads skyrockets
until we get
>>>>>>>>> OutOfMemoryError.
>>>>>>>>>
>>>>>>>>> It works just fine on regular hardware; We're going to
try VMWare
>>>>>>>>> tomorrow.
>>>>>>>>>
>>>>>>>>> One thing really worth mentioning is that by using the
fileCursor
>>>>>>>>> we
>>>>>>>>> actually started seeing it use the Temp Store.  When
reading about
>>>>>>>>> systemUsage it is NOT intuitive that the Temp Store does
not come
>>>>>>>>> into
>>>>>>>>> play
>>>>>>>>> with the default cursor.  Anyone keeping a significant
volume of
>>>>>>>>> messages on
>>>>>>>>> their queues should be well served by changing the cursor.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Mitch Granger wrote:
>>>>>>>>>
>>>>>>>>>  Config is attached.  We have also tried the
>>>>>>>>> activemq-scalability.xml
>>>>>>>>>> with
>>>>>>>>>> the only change being adding a stomp connector.
>>>>>>>>>>
>>>>>>>>>> Once we hit the memoryUsage limit we can [sometimes]
connect new
>>>>>>>>>> consumers
>>>>>>>>>> but nothing comes back after we send the SUBSCRIBE
frame.
>>>>>>>>>>
>>>>>>>>>> I expect sending to fail when we hit this limit but
if we can't
>>>>>>>>>> subscribe
>>>>>>>>>> there's no chance of recovering from this state.
>>>>>>>>>>
>>>>>>>>>> Rob Davies wrote:
>>>>>>>>>>
>>>>>>>>>>  On 26 Oct 2009, at 17:38, themitchy wrote:
>>>>>>>>>>>
>>>>>>>>>>>  We're using only persistent messages and heap
size is set to
>>>>>>>>>>> 2GB
>>>>>>>>>>> yet
>>>>>>>>>>> we
>>>>>>>>>>>
>>>>>>>>>>>> hit
>>>>>>>>>>>> the memoryUsage limit quite quickly (system
usage config
>>>>>>>>>>>> below).
>>>>>>>>>>>> This
>>>>>>>>>>>> is
>>>>>>>>>>>> followed by "java.lang.OutOfMemoryError:
unable to create new
>>>>>>>>>>>> native
>>>>>>>>>>>>  thread"
>>>>>>>>>>>> as the process quickly reaches the 2GB of
heap we gave it.  How
>>>>>>>>>>>> are
>>>>>>>>>>>> we
>>>>>>>>>>>> getting to that point with the memoryUsage
limit set far below
>>>>>>>>>>>> it?
>>>>>>>>>>>>
>>>>>>>>>>>> Is there no way to get AMQ to gracefully
limit it's memory
>>>>>>>>>>>> usage?
>>>>>>>>>>>>
>>>>>>>>>>>>      <systemUsage>
>>>>>>>>>>>>          <systemUsage>
>>>>>>>>>>>>              <memoryUsage>
>>>>>>>>>>>>                  <memoryUsage
limit="256 mb"/>
>>>>>>>>>>>>              </memoryUsage>
>>>>>>>>>>>>              <storeUsage>
>>>>>>>>>>>>                  <storeUsage
limit="60 gb" name="foo"/>
>>>>>>>>>>>>              </storeUsage>
>>>>>>>>>>>>              <tempUsage>
>>>>>>>>>>>>                  <tempUsage limit="60
gb"/>
>>>>>>>>>>>>              </tempUsage>
>>>>>>>>>>>>          </systemUsage>
>>>>>>>>>>>>      </systemUsage>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> View this message in context:
>>>>>>>>>>>> http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26064098.html
>>>>>>>>>>>> Sent from the ActiveMQ - User mailing list
archive at
>>>>>>>>>>>> Nabble.com.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>  Can you send the rest of your config ?
>>>>>>>>>>>
>>>>>>>>>>> Rob Davies
>>>>>>>>>>> http://twitter.com/rajdavies
>>>>>>>>>>> I work here: http://fusesource.com
>>>>>>>>>>> My Blog: http://rajdavies.blogspot.com/
>>>>>>>>>>> I'm writing this: http://www.manning.com/snyder/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>>
>>>> -----
>>>> Dejan Bosanac
>>>>
>>>> Open Source Integration - http://fusesource.com/
>>>> ActiveMQ in Action - http://www.manning.com/snyder/
>>>> Blog - http://www.nighttale.net
>>>>
>>> http://old.nabble.com/file/p26264779/dump.jpg
>>> --
>>> View this message in context:
>>> http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26264779.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>> 
>> 
>> 
>> -- 
>> http://blog.garytully.com
>> 
>> Open Source Integration
>> http://fusesource.com
>> 
>> 
>  http://old.nabble.com/file/p26278228/oom.jpg 
> 

-- 
View this message in context: http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26279671.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message