From users-return-20102-apmail-activemq-users-archive=activemq.apache.org@activemq.apache.org Thu Aug 13 12:53:52 2009 Return-Path: Delivered-To: apmail-activemq-users-archive@www.apache.org Received: (qmail 19110 invoked from network); 13 Aug 2009 12:53:51 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 13 Aug 2009 12:53:51 -0000 Received: (qmail 21882 invoked by uid 500); 13 Aug 2009 12:53:58 -0000 Delivered-To: apmail-activemq-users-archive@activemq.apache.org Received: (qmail 21810 invoked by uid 500); 13 Aug 2009 12:53:58 -0000 Mailing-List: contact users-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@activemq.apache.org Delivered-To: mailing list users@activemq.apache.org Received: (qmail 21800 invoked by uid 99); 13 Aug 2009 12:53:58 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 12:53:58 +0000 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.85.212.188] (HELO mail-vw0-f188.google.com) (209.85.212.188) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 12:53:46 +0000 Received: by vws26 with SMTP id 26so633343vws.26 for ; Thu, 13 Aug 2009 05:53:24 -0700 (PDT) MIME-Version: 1.0 Received: by 10.220.14.74 with SMTP id f10mr1113339vca.27.1250168004858; Thu, 13 Aug 2009 05:53:24 -0700 (PDT) In-Reply-To: <3a73c17c0908130217va53856bkb59381e11dfb51ff@mail.gmail.com> References: <5eda8340908121508o253d56eanacb5194759352f3e@mail.gmail.com> <3a73c17c0908130217va53856bkb59381e11dfb51ff@mail.gmail.com> Date: Thu, 13 Aug 2009 08:53:24 -0400 Message-ID: <5eda8340908130553u6b82584cw80b448b8c03c5e44@mail.gmail.com> Subject: Re: Broker deadlock in queue message delivery From: Chetan Sarva To: users@activemq.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org I think the blocked threads were all producers. This happened in a production system after about 6 days uptime, so I'm not sure yet if it is reproducible. I guess I'll find out in a few days :-) If my store usage limit is set to 1gb, then why would there be a problem when my data folder is less than 700mb? On Thu, Aug 13, 2009 at 5:17 AM, Gary Tully wrote: > Using jconsole you should be able to examine the store usage and size. > > Store disk space is released after a store checkpoint and cleanup and in > unreferenced maxDataFileSize chunks. It is not released on a per message = ack > basis. > > Do you have active consumers for the destinations that are waiting for > space? > > Can you reproduce this scenario? > > 2009/8/12 Chetan Sarva > >> Hi, >> >> We appear to be facing a deadlock during queue message delivery. See >> the following stack dump: >> >> "ActiveMQ Transport: tcp:///10.10.10.30:40919" daemon prio=3D10 >> tid=3D0x00002aab002a3800 nid=3D0x281e in Object.wait() >> [0x0000000057b77000] >> =A0 java.lang.Thread.State: TIMED_WAITING (on object monitor) >> =A0 =A0 =A0 =A0at java.lang.Object.wait(Native Method) >> =A0 =A0 =A0 =A0at org.apache.activemq.usage.Usage.waitForSpace(Usage.jav= a:99) >> =A0 =A0 =A0 =A0- locked <0x00002aaab37c9e68> (a java.lang.Object) >> =A0 =A0 =A0 =A0at >> org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:496) >> =A0 =A0 =A0 =A0- locked <0x00002aaab3a8cb48> (a java.lang.Object) >> =A0 =A0 =A0 =A0at org.apache.activemq.broker.region.Queue.send(Queue.jav= a:480) >> >> [...snip...] >> >> and several (15) of these: >> >> "ActiveMQ Transport: tcp:///10.10.10.31:55890" daemon prio=3D10 >> tid=3D0x00002aab00fbd800 nid=3D0x27f8 waiting for monitor entry >> [0x000000004f26a000] >> =A0 java.lang.Thread.State: BLOCKED (on object monitor) >> >> >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 at >> org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:489) >> >> >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0- waiting to >> lock <0x00002aaab3a8cb48> (a java.lang.Object) >> =A0 =A0 =A0 =A0at org.apache.activemq.broker.region.Queue.send(Queue.jav= a:480) >> >> >> We're using 5.3 (via fusesource) with the Kaha persistence adapter. >> Full broker config can be seen here - >> >> http://pastie.org/581968 >> >> It would seem from the stack trace above and from looking at the >> source, that it thinks the store usage is greater than 100% (just a >> guess) and is waiting for it to fall back to normal levels, however >> the entire data folder was only 630mb at the time of the deadlock. On >> restarting the server (had to kill -9 due to unresponsive threads) the >> data folder was no longer readable so I had to remove it (have a >> backup) in order to get the server started again. >> >> Any ideas as to what's going on here? >> >> regards, >> >> chetan >> > > > > -- > http://blog.garytully.com > > Open Source Integration > http://fusesource.com >