Return-Path: Delivered-To: apmail-activemq-users-archive@www.apache.org Received: (qmail 72274 invoked from network); 9 Nov 2009 15:02:17 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 9 Nov 2009 15:02:17 -0000 Received: (qmail 50343 invoked by uid 500); 9 Nov 2009 15:02:16 -0000 Delivered-To: apmail-activemq-users-archive@activemq.apache.org Received: (qmail 50281 invoked by uid 500); 9 Nov 2009 15:02:16 -0000 Mailing-List: contact users-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@activemq.apache.org Delivered-To: mailing list users@activemq.apache.org Received: (qmail 50271 invoked by uid 99); 9 Nov 2009 15:02:16 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Nov 2009 15:02:16 +0000 X-ASF-Spam-Status: No, hits=-2.6 required=5.0 tests=AWL,BAYES_00 X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of gary.tully@gmail.com designates 209.85.220.225 as permitted sender) Received: from [209.85.220.225] (HELO mail-fx0-f225.google.com) (209.85.220.225) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Nov 2009 15:02:13 +0000 Received: by fxm25 with SMTP id 25so1233069fxm.6 for ; Mon, 09 Nov 2009 07:01:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=9eb2BD1t36iz+77bFGuCLWBiMrxIZcO8hLWQKj1Bfqo=; b=PxWZ5CDsxhVeTyoyUf3Rx/k4h3Iwt0n/agDhfG+ZUWC2OuBn9WZGhfcBjXF6puEZRh 6LOiO6tza7LETJ1YFxZTKywKSO4ZIYfTFCakB7/pyvwRla/3c5Aszh/2ilWOfwS13UaS m9mLYdaNZWyUs5rEwpJQ02bheG+pwxo0fY1wU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=YU6ISFhxGZGo63R73XmjRSg4RgTbUnXzRn960V8Oo6mvdWLAkGgUTudsiAV8xJBtNv wBw4pV2duZ4E6XN+PfpALgTXWpTrGJazZNK0nLnyGhrhgplVnx5cindlo+dj6fm+ETLE xuDAlnu8tWQpNoyc86XUlWLTNs6zXpKgLeA2M= MIME-Version: 1.0 Received: by 10.204.7.195 with SMTP id e3mr688161bke.118.1257778911795; Mon, 09 Nov 2009 07:01:51 -0800 (PST) In-Reply-To: <26264779.post@talk.nabble.com> References: <26064098.post@talk.nabble.com> <60959CE2-9A5B-4900-964F-7815A87689BC@gmail.com> <4AE5F62B.9010501@sophos.com> <4AE639DE.5070000@sophos.com> <36e91d9d0910270527i7049384cmdb077d0f3b9d09ef@mail.gmail.com> <4AE765B8.6080100@sophos.com> <26093204.post@talk.nabble.com> <4AE87CDB.6020808@sophos.com> <36e91d9d0910281033j816a706j42139946cba5fbe0@mail.gmail.com> <26264779.post@talk.nabble.com> Date: Mon, 9 Nov 2009 15:01:51 +0000 Message-ID: <3a73c17c0911090701l2c80ab9ay4c9d92d529eeca3f@mail.gmail.com> Subject: Re: Out of Memory on 5.3 From: Gary Tully To: users@activemq.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable fyi: you can disable periodic message expiry processing using a destination policy entry that sets expireMessagesPeriod =3D 0 2009/11/9 afei : > > > when a large of messages in queue,and no consumer or the consumer is very > slow, the OOM problem occur, because : > in org.apache.activemq.broker.region.Queue,the 588 line is : > =A0doBrowse(true, browsedMessages, this.getMaxExpirePageSize()); > ,transform to : > doBrowse(false, browsedMessages, this.getMaxExpirePageSize()); > =A0is ok. > > > Dejan Bosanac wrote: >> >> Hi Mitch, >> >> yeah, I said in thread I was referring to, that it is working with >> "regular" >> stomp connector. I started investigating AMQ-2440 patch the other day, >> should be have something soon. >> >> Cheers >> -- >> Dejan Bosanac - http://twitter.com/dejanb >> >> Open Source Integration - http://fusesource.com/ >> ActiveMQ in Action - http://www.manning.com/snyder/ >> Blog - http://www.nighttale.net >> >> >> On Wed, Oct 28, 2009 at 6:18 PM, Mitch Granger >> wrote: >> >>> So we turned off stomp+nio and went back to plain old stomp and so far >>> it's >>> working fine. =A0New(IO) isn't always better, I guess :-) >>> >>> Seems like maybe it's this issue -> >>> https://issues.apache.org/activemq/browse/AMQ-2440 >>> >>> >>> afei wrote: >>> >>>> i have same problem >>>> >>>> http://www.nabble.com/file/p26093204/aaaaaa.jpg aaaaaa.jpg >>>> >>>> >>>> themitchy wrote: >>>> >>>>> This is what we've done to tune so far: >>>>> >>>>> =A0- UseDedicatedTaskRunner=3Dfalse >>>>> =A0- flow control is off >>>>> =A0- stomp transport uses transport.closeAsync=3Dfalse >>>>> >>>>> I agree that it is because of the high number of open/close connectio= ns >>>>> from Stomp. =A0When we monitor through JConsole we can see more threa= ds >>>>> starting up for each new connection. =A0The problem is that these thr= eads >>>>> don't get let go. =A0Even though the stomp clients are disconnecting = the >>>>> number of threads that get released is less than the number created. >>>>> So >>>>> =A0the thread count goes up and up until it fails. =A0All of the abov= e >>>>> settings/tuning only delay when it will hit the wall. >>>>> >>>>> Dejan Bosanac wrote: >>>>> >>>>>> Hi Mitch, >>>>>> >>>>>> I think the root cause of this problem is that you probably have Sto= mp >>>>>> clients that open/close connection at a high rate. I simulated this >>>>>> problem >>>>>> on OSX with a StompLoadTest ( >>>>>> >>>>>> http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/j= ava/org/apache/activemq/transport/stomp/StompLoadTest.java?view=3Dlog >>>>>> ), >>>>>> while trying to reproduce "too many open files" problem. You can fin= d >>>>>> some >>>>>> of my findings (and workaround) in this thread. >>>>>> >>>>>> >>>>>> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-S= tomp-tt25888831.html#a26010080 >>>>>> >>>>>> (BTW. it is producing "too many open files" problem on linux) >>>>>> Basically, >>>>>> the >>>>>> problem with stomp is that every send is done in separate connection >>>>>> and >>>>>> thus considered to be a new producer for every message. So when >>>>>> producer >>>>>> flow control is hit, the producers are piling up and probably not >>>>>> releasing >>>>>> connections. Thus you can observe large number of tcp connections on >>>>>> the >>>>>> system in state TIME_WAIT (and TIME_CLOSE), which causes that the >>>>>> system >>>>>> limit is hit at one point. In the above thread, you can find a >>>>>> workaround >>>>>> that worked for me for that test. I started investigating this more >>>>>> and >>>>>> hopefully I'll have some more findings in the near future. >>>>>> >>>>>> Cheers >>>>>> -- >>>>>> Dejan Bosanac - http://twitter.com/dejanb >>>>>> >>>>>> Open Source Integration - http://fusesource.com/ >>>>>> ActiveMQ in Action - http://www.manning.com/snyder/ >>>>>> Blog - http://www.nighttale.net >>>>>> >>>>>> >>>>>> On Tue, Oct 27, 2009 at 1:07 AM, Mitch Granger >>>>>> wrote: >>>>>> >>>>>> =A0Update: We've [nearly] proven that this only happens with AMQ run= ning >>>>>>> on >>>>>>> openVZ. =A0What exactly is causing it, we're still not sure. =A0Aft= er >>>>>>> memoryUsage is met, the number of threads skyrockets until we get >>>>>>> OutOfMemoryError. >>>>>>> >>>>>>> It works just fine on regular hardware; We're going to try VMWare >>>>>>> tomorrow. >>>>>>> >>>>>>> One thing really worth mentioning is that by using the fileCursor w= e >>>>>>> actually started seeing it use the Temp Store. =A0When reading abou= t >>>>>>> systemUsage it is NOT intuitive that the Temp Store does not come >>>>>>> into >>>>>>> play >>>>>>> with the default cursor. =A0Anyone keeping a significant volume of >>>>>>> messages on >>>>>>> their queues should be well served by changing the cursor. >>>>>>> >>>>>>> >>>>>>> Mitch Granger wrote: >>>>>>> >>>>>>> =A0Config is attached. =A0We have also tried the activemq-scalabili= ty.xml >>>>>>>> with >>>>>>>> the only change being adding a stomp connector. >>>>>>>> >>>>>>>> Once we hit the memoryUsage limit we can [sometimes] connect new >>>>>>>> consumers >>>>>>>> but nothing comes back after we send the SUBSCRIBE frame. >>>>>>>> >>>>>>>> I expect sending to fail when we hit this limit but if we can't >>>>>>>> subscribe >>>>>>>> there's no chance of recovering from this state. >>>>>>>> >>>>>>>> Rob Davies wrote: >>>>>>>> >>>>>>>> =A0On 26 Oct 2009, at 17:38, themitchy wrote: >>>>>>>>> >>>>>>>>> =A0We're using only persistent messages and heap size is set to 2= GB >>>>>>>>> yet >>>>>>>>> we >>>>>>>>> >>>>>>>>>> hit >>>>>>>>>> the memoryUsage limit quite quickly (system usage config below). >>>>>>>>>> This >>>>>>>>>> is >>>>>>>>>> followed by "java.lang.OutOfMemoryError: unable to create new >>>>>>>>>> native >>>>>>>>>> =A0thread" >>>>>>>>>> as the process quickly reaches the 2GB of heap we gave it. =A0Ho= w >>>>>>>>>> are >>>>>>>>>> we >>>>>>>>>> getting to that point with the memoryUsage limit set far below i= t? >>>>>>>>>> >>>>>>>>>> Is there no way to get AMQ to gracefully limit it's memory usage= ? >>>>>>>>>> >>>>>>>>>> =A0 =A0 =A0 >>>>>>>>>> =A0 =A0 =A0 =A0 =A0 >>>>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 >>>>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 >>>>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 >>>>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 >>>>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 >>>>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 >>>>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 >>>>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 >>>>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 >>>>>>>>>> =A0 =A0 =A0 =A0 =A0 >>>>>>>>>> =A0 =A0 =A0 >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> View this message in context: >>>>>>>>>> http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26064098.h= tml >>>>>>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com= . >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> =A0Can you send the rest of your config ? >>>>>>>>> >>>>>>>>> Rob Davies >>>>>>>>> http://twitter.com/rajdavies >>>>>>>>> I work here: http://fusesource.com >>>>>>>>> My Blog: http://rajdavies.blogspot.com/ >>>>>>>>> I'm writing this: http://www.manning.com/snyder/ >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> >>>> >> >> >> ----- >> Dejan Bosanac >> >> Open Source Integration - http://fusesource.com/ >> ActiveMQ in Action - http://www.manning.com/snyder/ >> Blog - http://www.nighttale.net >> > http://old.nabble.com/file/p26264779/dump.jpg > -- > View this message in context: http://old.nabble.com/Out-of-Memory-on-5.3-= tp26064098p26264779.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > > --=20 http://blog.garytully.com Open Source Integration http://fusesource.com