Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (nike.apache.org: domain of gary.tully@gmail.com designates
 209.85.220.225 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=v0q2J9QILdSnr5XWMPmRRy2vPtZC0CRyDyGjsolO2WSgnBr1WJRs/nm0MBGCvfxVoC
         I11/OolZVP56oEeRfGQVmWJ9JDdmQMe/BRbZmdcm5Cuo1u2eVzs9ztqHCjYavX6tGv4y
         YBuaQD59/yqlQ2Up/NM/BrilJQc/iYBFXqK18=
MIME-Version: 1.0
In-Reply-To: <26264779.post@talk.nabble.com>
References: <26064098.post@talk.nabble.com>
	 <60959CE2-9A5B-4900-964F-7815A87689BC@gmail.com>
	 <4AE5F62B.9010501@sophos.com> <4AE639DE.5070000@sophos.com>
	 <36e91d9d0910270527i7049384cmdb077d0f3b9d09ef@mail.gmail.com>
	 <4AE765B8.6080100@sophos.com> <26093204.post@talk.nabble.com>
	 <4AE87CDB.6020808@sophos.com>
	 <36e91d9d0910281033j816a706j42139946cba5fbe0@mail.gmail.com>
	 <26264779.post@talk.nabble.com>
Date: Mon, 9 Nov 2009 12:12:43 +0000
Message-ID: <3a73c17c0911090412g1969c926l716bb4bee84877ff@mail.gmail.com>
Subject: Re: Out of Memory on 5.3
From: Gary Tully <gary.tully@gmail.com>
To: users@activemq.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Think this is resolved by
https://issues.apache.org/activemq/browse/AMQ-2468 which is fixed on
trunk and the 5.3 branch. The pagedInPendingDispatch list which was
accumulating messages is now limited. This was highlighted by the jmx
purge operation but the same logic holds for the browse that is used
by there expiry task.
Can you validate with a current 5.4-SNAPSHOT?

2009/11/9 afei <afei1689@126.com>:
>
>
> when a large of messages in queue,and no consumer or the consumer is very
> slow, the OOM problem occur, because :
> in org.apache.activemq.broker.region.Queue,the 588 line is :
> =A0doBrowse(true, browsedMessages, this.getMaxExpirePageSize());
> ,transform to :
> doBrowse(false, browsedMessages, this.getMaxExpirePageSize());
> =A0is ok.
>
>
> Dejan Bosanac wrote:
>>
>> Hi Mitch,
>>
>> yeah, I said in thread I was referring to, that it is working with
>> "regular"
>> stomp connector. I started investigating AMQ-2440 patch the other day,
>> should be have something soon.
>>
>> Cheers
>> --
>> Dejan Bosanac - http://twitter.com/dejanb
>>
>> Open Source Integration - http://fusesource.com/
>> ActiveMQ in Action - http://www.manning.com/snyder/
>> Blog - http://www.nighttale.net
>>
>>
>> On Wed, Oct 28, 2009 at 6:18 PM, Mitch Granger
>> <mitch.granger@sophos.com>wrote:
>>
>>> So we turned off stomp+nio and went back to plain old stomp and so far
>>> it's
>>> working fine. =A0New(IO) isn't always better, I guess :-)
>>>
>>> Seems like maybe it's this issue ->
>>> https://issues.apache.org/activemq/browse/AMQ-2440
>>>
>>>
>>> afei wrote:
>>>
>>>> i have same problem
>>>>
>>>> http://www.nabble.com/file/p26093204/aaaaaa.jpg aaaaaa.jpg
>>>>
>>>>
>>>> themitchy wrote:
>>>>
>>>>> This is what we've done to tune so far:
>>>>>
>>>>> =A0- UseDedicatedTaskRunner=3Dfalse
>>>>> =A0- flow control is off
>>>>> =A0- stomp transport uses transport.closeAsync=3Dfalse
>>>>>
>>>>> I agree that it is because of the high number of open/close connectio=
ns
>>>>> from Stomp. =A0When we monitor through JConsole we can see more threa=
ds
>>>>> starting up for each new connection. =A0The problem is that these thr=
eads
>>>>> don't get let go. =A0Even though the stomp clients are disconnecting =
the
>>>>> number of threads that get released is less than the number created.
>>>>> So
>>>>> =A0the thread count goes up and up until it fails. =A0All of the abov=
e
>>>>> settings/tuning only delay when it will hit the wall.
>>>>>
>>>>> Dejan Bosanac wrote:
>>>>>
>>>>>> Hi Mitch,
>>>>>>
>>>>>> I think the root cause of this problem is that you probably have Sto=
mp
>>>>>> clients that open/close connection at a high rate. I simulated this
>>>>>> problem
>>>>>> on OSX with a StompLoadTest (
>>>>>>
>>>>>> http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/j=
ava/org/apache/activemq/transport/stomp/StompLoadTest.java?view=3Dlog
>>>>>> ),
>>>>>> while trying to reproduce "too many open files" problem. You can fin=
d
>>>>>> some
>>>>>> of my findings (and workaround) in this thread.
>>>>>>
>>>>>>
>>>>>> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-S=
tomp-tt25888831.html#a26010080
>>>>>>
>>>>>> (BTW. it is producing "too many open files" problem on linux)
>>>>>> Basically,
>>>>>> the
>>>>>> problem with stomp is that every send is done in separate connection
>>>>>> and
>>>>>> thus considered to be a new producer for every message. So when
>>>>>> producer
>>>>>> flow control is hit, the producers are piling up and probably not
>>>>>> releasing
>>>>>> connections. Thus you can observe large number of tcp connections on
>>>>>> the
>>>>>> system in state TIME_WAIT (and TIME_CLOSE), which causes that the
>>>>>> system
>>>>>> limit is hit at one point. In the above thread, you can find a
>>>>>> workaround
>>>>>> that worked for me for that test. I started investigating this more
>>>>>> and
>>>>>> hopefully I'll have some more findings in the near future.
>>>>>>
>>>>>> Cheers
>>>>>> --
>>>>>> Dejan Bosanac - http://twitter.com/dejanb
>>>>>>
>>>>>> Open Source Integration - http://fusesource.com/
>>>>>> ActiveMQ in Action - http://www.manning.com/snyder/
>>>>>> Blog - http://www.nighttale.net
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 27, 2009 at 1:07 AM, Mitch Granger
>>>>>> <mitch.granger@sophos.com>wrote:
>>>>>>
>>>>>> =A0Update: We've [nearly] proven that this only happens with AMQ run=
ning
>>>>>>> on
>>>>>>> openVZ. =A0What exactly is causing it, we're still not sure. =A0Aft=
er
>>>>>>> memoryUsage is met, the number of threads skyrockets until we get
>>>>>>> OutOfMemoryError.
>>>>>>>
>>>>>>> It works just fine on regular hardware; We're going to try VMWare
>>>>>>> tomorrow.
>>>>>>>
>>>>>>> One thing really worth mentioning is that by using the fileCursor w=
e
>>>>>>> actually started seeing it use the Temp Store. =A0When reading abou=
t
>>>>>>> systemUsage it is NOT intuitive that the Temp Store does not come
>>>>>>> into
>>>>>>> play
>>>>>>> with the default cursor. =A0Anyone keeping a significant volume of
>>>>>>> messages on
>>>>>>> their queues should be well served by changing the cursor.
>>>>>>>
>>>>>>>
>>>>>>> Mitch Granger wrote:
>>>>>>>
>>>>>>> =A0Config is attached. =A0We have also tried the activemq-scalabili=
ty.xml
>>>>>>>> with
>>>>>>>> the only change being adding a stomp connector.
>>>>>>>>
>>>>>>>> Once we hit the memoryUsage limit we can [sometimes] connect new
>>>>>>>> consumers
>>>>>>>> but nothing comes back after we send the SUBSCRIBE frame.
>>>>>>>>
>>>>>>>> I expect sending to fail when we hit this limit but if we can't
>>>>>>>> subscribe
>>>>>>>> there's no chance of recovering from this state.
>>>>>>>>
>>>>>>>> Rob Davies wrote:
>>>>>>>>
>>>>>>>> =A0On 26 Oct 2009, at 17:38, themitchy wrote:
>>>>>>>>>
>>>>>>>>> =A0We're using only persistent messages and heap size is set to 2=
GB
>>>>>>>>> yet
>>>>>>>>> we
>>>>>>>>>
>>>>>>>>>> hit
>>>>>>>>>> the memoryUsage limit quite quickly (system usage config below).
>>>>>>>>>> This
>>>>>>>>>> is
>>>>>>>>>> followed by "java.lang.OutOfMemoryError: unable to create new
>>>>>>>>>> native
>>>>>>>>>> =A0thread"
>>>>>>>>>> as the process quickly reaches the 2GB of heap we gave it. =A0Ho=
w
>>>>>>>>>> are
>>>>>>>>>> we
>>>>>>>>>> getting to that point with the memoryUsage limit set far below i=
t?
>>>>>>>>>>
>>>>>>>>>> Is there no way to get AMQ to gracefully limit it's memory usage=
?
>>>>>>>>>>
>>>>>>>>>> =A0 =A0 =A0<systemUsage>
>>>>>>>>>> =A0 =A0 =A0 =A0 =A0<systemUsage>
>>>>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0<memoryUsage>
>>>>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0<memoryUsage limit=3D"256 mb"=
/>
>>>>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0</memoryUsage>
>>>>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0<storeUsage>
>>>>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0<storeUsage limit=3D"60 gb" n=
ame=3D"foo"/>
>>>>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0</storeUsage>
>>>>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0<tempUsage>
>>>>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0<tempUsage limit=3D"60 gb"/>
>>>>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0</tempUsage>
>>>>>>>>>> =A0 =A0 =A0 =A0 =A0</systemUsage>
>>>>>>>>>> =A0 =A0 =A0</systemUsage>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> View this message in context:
>>>>>>>>>> http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26064098.h=
tml
>>>>>>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com=
.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> =A0Can you send the rest of your config ?
>>>>>>>>>
>>>>>>>>> Rob Davies
>>>>>>>>> http://twitter.com/rajdavies
>>>>>>>>> I work here: http://fusesource.com
>>>>>>>>> My Blog: http://rajdavies.blogspot.com/
>>>>>>>>> I'm writing this: http://www.manning.com/snyder/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>
>>>>
>>
>>
>> -----
>> Dejan Bosanac
>>
>> Open Source Integration - http://fusesource.com/
>> ActiveMQ in Action - http://www.manning.com/snyder/
>> Blog - http://www.nighttale.net
>>
> http://old.nabble.com/file/p26264779/dump.jpg
> --
> View this message in context: http://old.nabble.com/Out-of-Memory-on-5.3-=
tp26064098p26264779.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>


--=20
http://blog.garytully.com

Open Source Integration
http://fusesource.com