activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chetan Sarva <che...@milabra.com>
Subject Re: Broker deadlock in queue message delivery
Date Thu, 13 Aug 2009 12:53:24 GMT
I think the blocked threads were all producers. This happened in a
production system after about 6 days uptime, so I'm not sure yet if it
is reproducible. I guess I'll find out in a few days :-)

If my store usage limit is set to 1gb, then why would there be a
problem when my data folder is less than 700mb?

On Thu, Aug 13, 2009 at 5:17 AM, Gary Tully<gary.tully@gmail.com> wrote:
> Using jconsole you should be able to examine the store usage and size.
>
> Store disk space is released after a store checkpoint and cleanup and in
> unreferenced maxDataFileSize chunks. It is not released on a per message ack
> basis.
>
> Do you have active consumers for the destinations that are waiting for
> space?
>
> Can you reproduce this scenario?
>
> 2009/8/12 Chetan Sarva <chetan@milabra.com>
>
>> Hi,
>>
>> We appear to be facing a deadlock during queue message delivery. See
>> the following stack dump:
>>
>> "ActiveMQ Transport: tcp:///10.10.10.30:40919" daemon prio=10
>> tid=0x00002aab002a3800 nid=0x281e in Object.wait()
>> [0x0000000057b77000]
>>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>        at java.lang.Object.wait(Native Method)
>>        at org.apache.activemq.usage.Usage.waitForSpace(Usage.java:99)
>>        - locked <0x00002aaab37c9e68> (a java.lang.Object)
>>        at
>> org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:496)
>>        - locked <0x00002aaab3a8cb48> (a java.lang.Object)
>>        at org.apache.activemq.broker.region.Queue.send(Queue.java:480)
>>
>> [...snip...]
>>
>> and several (15) of these:
>>
>> "ActiveMQ Transport: tcp:///10.10.10.31:55890" daemon prio=10
>> tid=0x00002aab00fbd800 nid=0x27f8 waiting for monitor entry
>> [0x000000004f26a000]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>
>>
>>                                                      
          at
>> org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:489)
>>
>>
>>                                                      -
waiting to
>> lock <0x00002aaab3a8cb48> (a java.lang.Object)
>>        at org.apache.activemq.broker.region.Queue.send(Queue.java:480)
>>
>>
>> We're using 5.3 (via fusesource) with the Kaha persistence adapter.
>> Full broker config can be seen here -
>>
>> http://pastie.org/581968
>>
>> It would seem from the stack trace above and from looking at the
>> source, that it thinks the store usage is greater than 100% (just a
>> guess) and is waiting for it to fall back to normal levels, however
>> the entire data folder was only 630mb at the time of the deadlock. On
>> restarting the server (had to kill -9 due to unresponsive threads) the
>> data folder was no longer readable so I had to remove it (have a
>> backup) in order to get the server started again.
>>
>> Any ideas as to what's going on here?
>>
>> regards,
>>
>> chetan
>>
>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
>

Mime
View raw message