activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <gary.tu...@gmail.com>
Subject Re: Broker deadlock in queue message delivery
Date Thu, 13 Aug 2009 09:17:33 GMT
Using jconsole you should be able to examine the store usage and size.

Store disk space is released after a store checkpoint and cleanup and in
unreferenced maxDataFileSize chunks. It is not released on a per message ack
basis.

Do you have active consumers for the destinations that are waiting for
space?

Can you reproduce this scenario?

2009/8/12 Chetan Sarva <chetan@milabra.com>

> Hi,
>
> We appear to be facing a deadlock during queue message delivery. See
> the following stack dump:
>
> "ActiveMQ Transport: tcp:///10.10.10.30:40919" daemon prio=10
> tid=0x00002aab002a3800 nid=0x281e in Object.wait()
> [0x0000000057b77000]
>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        at org.apache.activemq.usage.Usage.waitForSpace(Usage.java:99)
>        - locked <0x00002aaab37c9e68> (a java.lang.Object)
>        at
> org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:496)
>        - locked <0x00002aaab3a8cb48> (a java.lang.Object)
>        at org.apache.activemq.broker.region.Queue.send(Queue.java:480)
>
> [...snip...]
>
> and several (15) of these:
>
> "ActiveMQ Transport: tcp:///10.10.10.31:55890" daemon prio=10
> tid=0x00002aab00fbd800 nid=0x27f8 waiting for monitor entry
> [0x000000004f26a000]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>
>
>                                                                 at
> org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:489)
>
>
>                                                      - waiting to
> lock <0x00002aaab3a8cb48> (a java.lang.Object)
>        at org.apache.activemq.broker.region.Queue.send(Queue.java:480)
>
>
> We're using 5.3 (via fusesource) with the Kaha persistence adapter.
> Full broker config can be seen here -
>
> http://pastie.org/581968
>
> It would seem from the stack trace above and from looking at the
> source, that it thinks the store usage is greater than 100% (just a
> guess) and is waiting for it to fall back to normal levels, however
> the entire data folder was only 630mb at the time of the deadlock. On
> restarting the server (had to kill -9 due to unresponsive threads) the
> data folder was no longer readable so I had to remove it (have a
> backup) in order to get the server started again.
>
> Any ideas as to what's going on here?
>
> regards,
>
> chetan
>



-- 
http://blog.garytully.com

Open Source Integration
http://fusesource.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message