Return-Path: X-Original-To: apmail-activemq-dev-archive@www.apache.org Delivered-To: apmail-activemq-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 891A7D695 for ; Fri, 21 Sep 2012 23:02:08 +0000 (UTC) Received: (qmail 22397 invoked by uid 500); 21 Sep 2012 23:02:08 -0000 Delivered-To: apmail-activemq-dev-archive@activemq.apache.org Received: (qmail 22354 invoked by uid 500); 21 Sep 2012 23:02:08 -0000 Mailing-List: contact dev-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@activemq.apache.org Delivered-To: mailing list dev@activemq.apache.org Received: (qmail 22345 invoked by uid 99); 21 Sep 2012 23:02:08 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Sep 2012 23:02:08 +0000 Date: Sat, 22 Sep 2012 10:02:08 +1100 (NCT) From: "Claudio Santana (JIRA)" To: dev@activemq.apache.org Message-ID: <1724647499.110063.1348268528095.JavaMail.jiratomcat@arcas> Subject: [jira] [Created] (AMQ-4071) Restarting of persistent embedded broker causing Failed to fill batch error MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Claudio Santana created AMQ-4071: ------------------------------------ Summary: Restarting of persistent embedded broker causing Failed to fill batch error Key: AMQ-4071 URL: https://issues.apache.org/jira/browse/AMQ-4071 Project: ActiveMQ Issue Type: Bug Components: Broker Affects Versions: 5.6.0 Environment: Ubuntu 12.04.1 LTS java version "1.6.0_32" Java(TM) SE Runtime Environment (build 1.6.0_32-b05) Java HotSpot(TM) 64-Bit Server VM (build 20.7-b02, mixed mode) Tomcat 6.0.35 Reporter: Claudio Santana Priority: Critical I have a Tomcat web container with a persistent embedded broker in it, all messages get produced in this web container and queued up in the embedded broker. The embedded broker connects to a stand alone master/slave shared storage broker and finally the messages get consumed by threads clients connecting directly to the stand alone broker. The consumers run inside a Tomcat web container of the same version. While the stand alone broker is enqueueing about 17 messages a second I restart the message producers which have embedded brokers the exception bellow starts appearing in the stand alone broker log: INFO | jvm 1 | 2012/09/21 22:00:32 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@40b1d29:myqueue,batchResetNeeded=false,storeHasMessages=true,size=30650,cacheEnabled=false - Failed to fill batch INFO | jvm 1 | 2012/09/21 22:00:32 | java.lang.RuntimeException: java.lang.ClassCastException: org.apache.activemq.store.kahadb.data.KahaTraceCommand cannot be cast to org.apache.activemq.store.kahadb.data.KahaAddMessageCommand INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:280) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1766) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1995) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1488) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36) INFO | jvm 1 | 2012/09/21 22:00:32 | Caused by: java.lang.ClassCastException: org.apache.activemq.store.kahadb.data.KahaTraceCommand cannot be cast to org.apache.activemq.store.kahadb.data.KahaAddMessageCommand INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:1015) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:556) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.kahadb.page.Transaction.execute(Transaction.java:769) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:545) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:106) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:277) INFO | jvm 1 | 2012/09/21 22:00:32 | ... 7 more INFO | jvm 1 | 2012/09/21 22:00:32 | ERROR | Failed to page in more queue messages INFO | jvm 1 | 2012/09/21 22:00:32 | java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassCastException: org.apache.activemq.store.kahadb.data.KahaTraceCommand cannot be cast to org.apache.activemq.store.kahadb.data.KahaAddMessageCommand INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:116) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1766) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1995) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1488) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36) INFO | jvm 1 | 2012/09/21 22:00:32 | Caused by: java.lang.RuntimeException: java.lang.ClassCastException: org.apache.activemq.store.kahadb.data.KahaTraceCommand cannot be cast to org.apache.activemq.store.kahadb.data.KahaAddMessageCommand INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:280) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113) INFO | jvm 1 | 2012/09/21 22:00:32 | ... 6 more INFO | jvm 1 | 2012/09/21 22:00:32 | Caused by: java.lang.ClassCastException: org.apache.activemq.store.kahadb.data.KahaTraceCommand cannot be cast to org.apache.activemq.store.kahadb.data.KahaAddMessageCommand INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:1015) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:556) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.kahadb.page.Transaction.execute(Transaction.java:769) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:545) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:106) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97) INFO | jvm 1 | 2012/09/21 22:00:32 | at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:277) INFO | jvm 1 | 2012/09/21 22:00:32 | ... 7 more I have 5 different queues in my brokers and I don't see any issues with any of the other queues. I've experienced this issue before and some other queue was the one affected. These servers were being used at about 5 messages per second, they would be regularly restarted and this issue never appeared before. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira