Hi All,
We experienced a major outage yesterday where ActveMQ 5.5 broker crashed and
stopped accepting any connections from the client. Please note that broker
process was still running. Below is the error that we got from the client end.
"Connection refused at /usr/local/pkgs/site_perl-5.10/lib/Net/Stomp.pm line 19"
>From the broker logs, I can see that it started spitting out below errors around
that time:
6 | ERROR | Could not accept connection : java.lang.IllegalStateException: Timer
already cancelled. | org.apache.activemq.broker.TransportConnector | ActiveMQ
Task-241791
2011-11-24 16:51:39,582 | ERROR | Could not accept connection :
java.lang.IllegalStateException: Timer already cancelled. |
org.apache.activemq.broker.TransportConnector | ActiveMQ Task-241876
2011-11-24 16:51:46,365 | ERROR | Could not accept connection :
java.lang.IllegalStateException: Timer already cancelled. |
org.apache.activemq.broker.TransportConnector | ActiveMQ Task-241889
2011-11-24 16:52:10,114 | ERROR | Could not accept connection :
java.lang.IllegalStateException: Timer already cancelled. |
org.apache.activemq.broker.TransportConnector | ActiveMQ Task-241914
Before these errors came up I can see that there are info level messages but
these we have been getting almost daily:
011-11-24 00:11:29,492 | INFO | Transport failed:
org.apache.activemq.transport.InactivityIOException: Channel was inactive for
too (>30000) long: /10.77.73.176:54410 |
org.apache.activemq.broker.TransportConnection.Transport | InactivityMonitor
Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@10b8b87
011-11-24 00:11:29,492 | INFO | Transport failed:
org.apache.activemq.transport.InactivityIOException: Channel was inactive for
too (>30000) long: /10.77.73.176:54410 |
org.apache.activemq.broker.TransportConnection.Transport | InactivityMonitor
Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@10b8b87
>From the jconsole only suspicious thing that we could find that is the PS
Survivor memory space had a spike of 80 MB during that time.
Could someone please help us here figure what could have been the root cause.
Any help would be appreciated.
Thanks
|