Return-Path: X-Original-To: apmail-activemq-users-archive@www.apache.org Delivered-To: apmail-activemq-users-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 21A11D47E for ; Wed, 19 Sep 2012 19:56:07 +0000 (UTC) Received: (qmail 1291 invoked by uid 500); 19 Sep 2012 19:56:06 -0000 Delivered-To: apmail-activemq-users-archive@activemq.apache.org Received: (qmail 1253 invoked by uid 500); 19 Sep 2012 19:56:06 -0000 Mailing-List: contact users-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@activemq.apache.org Delivered-To: mailing list users@activemq.apache.org Received: (qmail 1180 invoked by uid 99); 19 Sep 2012 19:56:05 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Sep 2012 19:56:05 +0000 X-ASF-Spam-Status: No, hits=2.0 required=5.0 tests=SPF_NEUTRAL,URI_HEX X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: 216.139.250.139 is neither permitted nor denied by domain of claudio.santana@gmail.com) Received: from [216.139.250.139] (HELO joe.nabble.com) (216.139.250.139) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Sep 2012 19:55:59 +0000 Received: from [192.168.236.139] (helo=joe.nabble.com) by joe.nabble.com with esmtp (Exim 4.72) (envelope-from ) id 1TEQN0-0000h2-SM for users@activemq.apache.org; Wed, 19 Sep 2012 12:55:38 -0700 Date: Wed, 19 Sep 2012 12:55:38 -0700 (PDT) From: pollotek To: users@activemq.apache.org Message-ID: <1348084538868-4656715.post@n4.nabble.com> In-Reply-To: <1348010430389-4656612.post@n4.nabble.com> References: <1348010430389-4656612.post@n4.nabble.com> Subject: Re: Broker stuck showing error messages in log MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Based on some other post I decided to add the following parameters to my configuration: The system seem to have started correctly but after a few minutes util it started throwing the following exceptions: INFO | jvm 1 | 2012/09/19 00:43:12 | INFO | Recovering from the journal ... INFO | jvm 1 | 2012/09/19 00:43:12 | INFO | Recovery replayed 1 operations from the journal in 0.017 seconds. INFO | jvm 1 | 2012/09/19 00:43:12 | INFO | ActiveMQ 5.6.0 JMS Message Broker (broker) is starting INFO | jvm 1 | 2012/09/19 00:43:12 | INFO | For help or more information please see: http://activemq.apache.org/ INFO | jvm 1 | 2012/09/19 00:43:16 | INFO | Listening for connections at: tcp://mqbroker02.fitbit.com:61616 INFO | jvm 1 | 2012/09/19 00:43:16 | INFO | Connector openwire Started INFO | jvm 1 | 2012/09/19 00:43:16 | INFO | ActiveMQ JMS Message Broker (broker, ID:mqbroker02-57219-1348015392398-0:1) started INFO | jvm 1 | 2012/09/19 00:43:21 | WARN | Duplicate message add attempt rejected. Destination: com.fitbit.queue.mixpanelNotification, Message id: ID:app4.fit bit.com-56277-1348008881548-5:2:5273:1:1 INFO | jvm 1 | 2012/09/19 00:43:26 | WARN | Duplicate message add attempt rejected. Destination: com.fitbit.queue.timedData, Message id: ID:app7.fitbit.com-451 37-1348003510612-5:2:1381:1:1 INFO | jvm 1 | 2012/09/19 00:43:31 | WARN | Duplicate message add attempt rejected. Destination: com.fitbit.queue.timedData, Message id: ID:app11-46276-1348004 200281-5:2:3293:1:1 INFO | jvm 1 | 2012/09/19 00:43:32 | WARN | Duplicate message add attempt rejected. Destination: com.fitbit.queue.mixpanelNotification, Message id: ID:app10.fi tbit.com-49543-1348009973081-5:2:8111:1:1 INFO | jvm 1 | 2012/09/19 00:43:33 | WARN | Duplicate message add attempt rejected. Destination: com.fitbit.queue.timedData, Message id: ID:app12-47368-1348010 402189-5:2:6151:1:1 INFO | jvm 1 | 2012/09/19 00:43:35 | WARN | Duplicate message add attempt rejected. Destination: com.fitbit.queue.mixpanelNotification, Message id: ID:app3.fit bit.com-33478-1348008694249-5:2:5384:1:1 INFO | jvm 1 | 2012/09/19 00:46:53 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@2f012501:com.fitbit.queue.timedData,batchResetNeeded=f alse,storeHasMessages=true,size=4,cacheEnabled=false - Failed to fill batch INFO | jvm 1 | 2012/09/19 00:46:53 | java.lang.ClassCastException: org.apache.activemq.store.kahadb.data.KahaTraceCommand cannot be cast to org.apache.activemq. store.kahadb.data.KahaAddMessageCommand INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:1015) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:556) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.kahadb.page.Transaction.execute(Transaction.java:769) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:545) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:106) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:277) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1766) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1995) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1488) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36) INFO | jvm 1 | 2012/09/19 00:46:53 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@2f012501:com.fitbit.queue.timedData,batchResetNeeded=f alse,storeHasMessages=true,size=4,cacheEnabled=false - Failed to fill batch INFO | jvm 1 | 2012/09/19 00:46:53 | java.lang.RuntimeException: java.lang.ClassCastException: org.apache.activemq.store.kahadb.data.KahaTraceCommand cannot be cast to org.apache.activemq.store.kahadb.data.KahaAddMessageCommand INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:280) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1766) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1995) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1488) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36) INFO | jvm 1 | 2012/09/19 00:46:53 | Caused by: java.lang.ClassCastException: org.apache.activemq.store.kahadb.data.KahaTraceCommand cannot be cast to org.apach e.activemq.store.kahadb.data.KahaAddMessageCommand INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:1015) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:556) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.kahadb.page.Transaction.execute(Transaction.java:769) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:545) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:106) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:277) INFO | jvm 1 | 2012/09/19 00:46:53 | ... 7 more INFO | jvm 1 | 2012/09/19 00:46:53 | ERROR | Failed to page in more queue messages INFO | jvm 1 | 2012/09/19 00:46:53 | java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassCastException: org.apache.activemq.store.kahadb.data.KahaTraceCommand cannot be cast to org.apache.activemq.store.kahadb.data.KahaAddMessageCommand INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:116) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1766) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1995) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1488) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36) INFO | jvm 1 | 2012/09/19 00:46:53 | Caused by: java.lang.RuntimeException: java.lang.ClassCastException: org.apache.activemq.store.kahadb.data.KahaTraceCommand cannot be cast to org.apache.activemq.store.kahadb.data.KahaAddMessageCommand INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:280) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113) INFO | jvm 1 | 2012/09/19 00:46:53 | ... 6 more INFO | jvm 1 | 2012/09/19 00:46:53 | Caused by: java.lang.ClassCastException: org.apache.activemq.store.kahadb.data.KahaTraceCommand cannot be cast to org.apache.activemq.store.kahadb.data.KahaAddMessageCommand INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:1015) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:556) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.kahadb.page.Transaction.execute(Transaction.java:769) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:545) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:106) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97) INFO | jvm 1 | 2012/09/19 00:46:53 | at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:277) INFO | jvm 1 | 2012/09/19 00:46:53 | ... 7 more This persistent stand alone broker has messages coming form embedded brokers and directly connected consumers. The message producing embedded brokers are still flowing to the stand alone broker, I can see the size of the queue grow but the consumers (I have 20 threads) are not getting any messages nor throwing any exceptions. Some help would be really appreciated. -- View this message in context: http://activemq.2283324.n4.nabble.com/Broker-stuck-showing-error-messages-in-log-tp4656612p4656715.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.