Return-Path: X-Original-To: apmail-activemq-users-archive@www.apache.org Delivered-To: apmail-activemq-users-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E3DB57715 for ; Fri, 25 Nov 2011 07:30:50 +0000 (UTC) Received: (qmail 37128 invoked by uid 500); 25 Nov 2011 07:30:50 -0000 Delivered-To: apmail-activemq-users-archive@activemq.apache.org Received: (qmail 36951 invoked by uid 500); 25 Nov 2011 07:30:44 -0000 Mailing-List: contact users-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@activemq.apache.org Delivered-To: mailing list users@activemq.apache.org Received: (qmail 36937 invoked by uid 99); 25 Nov 2011 07:30:39 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Nov 2011 07:30:39 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of gcjau-user-2@m.gmane.org designates 80.91.229.12 as permitted sender) Received: from [80.91.229.12] (HELO lo.gmane.org) (80.91.229.12) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Nov 2011 07:30:30 +0000 Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1RTqEY-0003fS-0S for users@activemq.apache.org; Fri, 25 Nov 2011 08:30:06 +0100 Received: from alevin.hyd.deshaw.com ([149.77.176.209]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 25 Nov 2011 08:30:05 +0100 Received: from vivek-gupta by alevin.hyd.deshaw.com with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 25 Nov 2011 08:30:05 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: users@activemq.apache.org From: ActiveMQ user Subject: ActiveMQ 5.5 broker stopped accepting connections Date: Fri, 25 Nov 2011 07:27:46 +0000 (UTC) Lines: 52 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: sea.gmane.org User-Agent: Loom/3.14 (http://gmane.org/) X-Loom-IP: 149.77.176.209 (Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.120 Safari/535.2) X-Virus-Checked: Checked by ClamAV on apache.org Hi All, We experienced a major outage yesterday where ActveMQ 5.5 broker crashed and stopped accepting any connections from the client. Please note that broker process was still running. Below is the error that we got from the client end. "Connection refused at /usr/local/pkgs/site_perl-5.10/lib/Net/Stomp.pm line 19" >From the broker logs, I can see that it started spitting out below errors around that time: 6 | ERROR | Could not accept connection : java.lang.IllegalStateException: Timer already cancelled. | org.apache.activemq.broker.TransportConnector | ActiveMQ Task-241791 2011-11-24 16:51:39,582 | ERROR | Could not accept connection : java.lang.IllegalStateException: Timer already cancelled. | org.apache.activemq.broker.TransportConnector | ActiveMQ Task-241876 2011-11-24 16:51:46,365 | ERROR | Could not accept connection : java.lang.IllegalStateException: Timer already cancelled. | org.apache.activemq.broker.TransportConnector | ActiveMQ Task-241889 2011-11-24 16:52:10,114 | ERROR | Could not accept connection : java.lang.IllegalStateException: Timer already cancelled. | org.apache.activemq.broker.TransportConnector | ActiveMQ Task-241914 Before these errors came up I can see that there are info level messages but these we have been getting almost daily: 011-11-24 00:11:29,492 | INFO | Transport failed: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>30000) long: /10.77.73.176:54410 | org.apache.activemq.broker.TransportConnection.Transport | InactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@10b8b87 011-11-24 00:11:29,492 | INFO | Transport failed: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>30000) long: /10.77.73.176:54410 | org.apache.activemq.broker.TransportConnection.Transport | InactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@10b8b87 >From the jconsole only suspicious thing that we could find that is the PS Survivor memory space had a spike of 80 MB during that time. Could someone please help us here figure what could have been the root cause. Any help would be appreciated. Thanks