activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From afei <afei1...@126.com>
Subject Re: Out of Memory on 5.3
Date Tue, 10 Nov 2009 04:45:14 GMT

in addition,another problem of OOM.



Gary Tully wrote:
> 
> fyi: you can disable periodic message expiry processing using a
> destination policy entry that sets expireMessagesPeriod = 0
> 
> 2009/11/9 afei <afei1689@126.com>:
>>
>>
>> when a large of messages in queue,and no consumer or the consumer is very
>> slow, the OOM problem occur, because :
>> in org.apache.activemq.broker.region.Queue,the 588 line is :
>>  doBrowse(true, browsedMessages, this.getMaxExpirePageSize());
>> ,transform to :
>> doBrowse(false, browsedMessages, this.getMaxExpirePageSize());
>>  is ok.
>>
>>
>> Dejan Bosanac wrote:
>>>
>>> Hi Mitch,
>>>
>>> yeah, I said in thread I was referring to, that it is working with
>>> "regular"
>>> stomp connector. I started investigating AMQ-2440 patch the other day,
>>> should be have something soon.
>>>
>>> Cheers
>>> --
>>> Dejan Bosanac - http://twitter.com/dejanb
>>>
>>> Open Source Integration - http://fusesource.com/
>>> ActiveMQ in Action - http://www.manning.com/snyder/
>>> Blog - http://www.nighttale.net
>>>
>>>
>>> On Wed, Oct 28, 2009 at 6:18 PM, Mitch Granger
>>> <mitch.granger@sophos.com>wrote:
>>>
>>>> So we turned off stomp+nio and went back to plain old stomp and so far
>>>> it's
>>>> working fine.  New(IO) isn't always better, I guess :-)
>>>>
>>>> Seems like maybe it's this issue ->
>>>> https://issues.apache.org/activemq/browse/AMQ-2440
>>>>
>>>>
>>>> afei wrote:
>>>>
>>>>> i have same problem
>>>>>
>>>>> http://www.nabble.com/file/p26093204/aaaaaa.jpg aaaaaa.jpg
>>>>>
>>>>>
>>>>> themitchy wrote:
>>>>>
>>>>>> This is what we've done to tune so far:
>>>>>>
>>>>>>  - UseDedicatedTaskRunner=false
>>>>>>  - flow control is off
>>>>>>  - stomp transport uses transport.closeAsync=false
>>>>>>
>>>>>> I agree that it is because of the high number of open/close
>>>>>> connections
>>>>>> from Stomp.  When we monitor through JConsole we can see more threads
>>>>>> starting up for each new connection.  The problem is that these
>>>>>> threads
>>>>>> don't get let go.  Even though the stomp clients are disconnecting
>>>>>> the
>>>>>> number of threads that get released is less than the number created.
>>>>>> So
>>>>>>  the thread count goes up and up until it fails.  All of the above
>>>>>> settings/tuning only delay when it will hit the wall.
>>>>>>
>>>>>> Dejan Bosanac wrote:
>>>>>>
>>>>>>> Hi Mitch,
>>>>>>>
>>>>>>> I think the root cause of this problem is that you probably have
>>>>>>> Stomp
>>>>>>> clients that open/close connection at a high rate. I simulated
this
>>>>>>> problem
>>>>>>> on OSX with a StompLoadTest (
>>>>>>>
>>>>>>> http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/transport/stomp/StompLoadTest.java?view=log
>>>>>>> ),
>>>>>>> while trying to reproduce "too many open files" problem. You
can
>>>>>>> find
>>>>>>> some
>>>>>>> of my findings (and workaround) in this thread.
>>>>>>>
>>>>>>>
>>>>>>> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tt25888831.html#a26010080
>>>>>>>
>>>>>>> (BTW. it is producing "too many open files" problem on linux)
>>>>>>> Basically,
>>>>>>> the
>>>>>>> problem with stomp is that every send is done in separate connection
>>>>>>> and
>>>>>>> thus considered to be a new producer for every message. So when
>>>>>>> producer
>>>>>>> flow control is hit, the producers are piling up and probably
not
>>>>>>> releasing
>>>>>>> connections. Thus you can observe large number of tcp connections
on
>>>>>>> the
>>>>>>> system in state TIME_WAIT (and TIME_CLOSE), which causes that
the
>>>>>>> system
>>>>>>> limit is hit at one point. In the above thread, you can find
a
>>>>>>> workaround
>>>>>>> that worked for me for that test. I started investigating this
more
>>>>>>> and
>>>>>>> hopefully I'll have some more findings in the near future.
>>>>>>>
>>>>>>> Cheers
>>>>>>> --
>>>>>>> Dejan Bosanac - http://twitter.com/dejanb
>>>>>>>
>>>>>>> Open Source Integration - http://fusesource.com/
>>>>>>> ActiveMQ in Action - http://www.manning.com/snyder/
>>>>>>> Blog - http://www.nighttale.net
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Oct 27, 2009 at 1:07 AM, Mitch Granger
>>>>>>> <mitch.granger@sophos.com>wrote:
>>>>>>>
>>>>>>>  Update: We've [nearly] proven that this only happens with AMQ
>>>>>>> running
>>>>>>>> on
>>>>>>>> openVZ.  What exactly is causing it, we're still not sure.
 After
>>>>>>>> memoryUsage is met, the number of threads skyrockets until
we get
>>>>>>>> OutOfMemoryError.
>>>>>>>>
>>>>>>>> It works just fine on regular hardware; We're going to try
VMWare
>>>>>>>> tomorrow.
>>>>>>>>
>>>>>>>> One thing really worth mentioning is that by using the fileCursor
>>>>>>>> we
>>>>>>>> actually started seeing it use the Temp Store.  When reading
about
>>>>>>>> systemUsage it is NOT intuitive that the Temp Store does
not come
>>>>>>>> into
>>>>>>>> play
>>>>>>>> with the default cursor.  Anyone keeping a significant volume
of
>>>>>>>> messages on
>>>>>>>> their queues should be well served by changing the cursor.
>>>>>>>>
>>>>>>>>
>>>>>>>> Mitch Granger wrote:
>>>>>>>>
>>>>>>>>  Config is attached.  We have also tried the
>>>>>>>> activemq-scalability.xml
>>>>>>>>> with
>>>>>>>>> the only change being adding a stomp connector.
>>>>>>>>>
>>>>>>>>> Once we hit the memoryUsage limit we can [sometimes]
connect new
>>>>>>>>> consumers
>>>>>>>>> but nothing comes back after we send the SUBSCRIBE frame.
>>>>>>>>>
>>>>>>>>> I expect sending to fail when we hit this limit but if
we can't
>>>>>>>>> subscribe
>>>>>>>>> there's no chance of recovering from this state.
>>>>>>>>>
>>>>>>>>> Rob Davies wrote:
>>>>>>>>>
>>>>>>>>>  On 26 Oct 2009, at 17:38, themitchy wrote:
>>>>>>>>>>
>>>>>>>>>>  We're using only persistent messages and heap size
is set to 2GB
>>>>>>>>>> yet
>>>>>>>>>> we
>>>>>>>>>>
>>>>>>>>>>> hit
>>>>>>>>>>> the memoryUsage limit quite quickly (system usage
config below).
>>>>>>>>>>> This
>>>>>>>>>>> is
>>>>>>>>>>> followed by "java.lang.OutOfMemoryError: unable
to create new
>>>>>>>>>>> native
>>>>>>>>>>>  thread"
>>>>>>>>>>> as the process quickly reaches the 2GB of heap
we gave it.  How
>>>>>>>>>>> are
>>>>>>>>>>> we
>>>>>>>>>>> getting to that point with the memoryUsage limit
set far below
>>>>>>>>>>> it?
>>>>>>>>>>>
>>>>>>>>>>> Is there no way to get AMQ to gracefully limit
it's memory
>>>>>>>>>>> usage?
>>>>>>>>>>>
>>>>>>>>>>>      <systemUsage>
>>>>>>>>>>>          <systemUsage>
>>>>>>>>>>>              <memoryUsage>
>>>>>>>>>>>                  <memoryUsage limit="256
mb"/>
>>>>>>>>>>>              </memoryUsage>
>>>>>>>>>>>              <storeUsage>
>>>>>>>>>>>                  <storeUsage limit="60
gb" name="foo"/>
>>>>>>>>>>>              </storeUsage>
>>>>>>>>>>>              <tempUsage>
>>>>>>>>>>>                  <tempUsage limit="60
gb"/>
>>>>>>>>>>>              </tempUsage>
>>>>>>>>>>>          </systemUsage>
>>>>>>>>>>>      </systemUsage>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> View this message in context:
>>>>>>>>>>> http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26064098.html
>>>>>>>>>>> Sent from the ActiveMQ - User mailing list archive
at
>>>>>>>>>>> Nabble.com.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  Can you send the rest of your config ?
>>>>>>>>>>
>>>>>>>>>> Rob Davies
>>>>>>>>>> http://twitter.com/rajdavies
>>>>>>>>>> I work here: http://fusesource.com
>>>>>>>>>> My Blog: http://rajdavies.blogspot.com/
>>>>>>>>>> I'm writing this: http://www.manning.com/snyder/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>>>>
>>>
>>>
>>> -----
>>> Dejan Bosanac
>>>
>>> Open Source Integration - http://fusesource.com/
>>> ActiveMQ in Action - http://www.manning.com/snyder/
>>> Blog - http://www.nighttale.net
>>>
>> http://old.nabble.com/file/p26264779/dump.jpg
>> --
>> View this message in context:
>> http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26264779.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
> 
> 
> 
> -- 
> http://blog.garytully.com
> 
> Open Source Integration
> http://fusesource.com
> 
> 
http://old.nabble.com/file/p26278228/oom.jpg 
-- 
View this message in context: http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26278228.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message