Return-Path: X-Original-To: apmail-activemq-dev-archive@www.apache.org Delivered-To: apmail-activemq-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 14A1D101FD for ; Tue, 25 Nov 2014 21:15:13 +0000 (UTC) Received: (qmail 76620 invoked by uid 500); 25 Nov 2014 21:15:12 -0000 Delivered-To: apmail-activemq-dev-archive@activemq.apache.org Received: (qmail 76532 invoked by uid 500); 25 Nov 2014 21:15:12 -0000 Mailing-List: contact dev-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@activemq.apache.org Delivered-To: mailing list dev@activemq.apache.org Received: (qmail 76205 invoked by uid 99); 25 Nov 2014 21:15:12 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Nov 2014 21:15:12 +0000 Date: Tue, 25 Nov 2014 21:15:12 +0000 (UTC) From: "Pankaj Takawale (JIRA)" To: dev@activemq.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (AMQ-5459) leveldb failed to page in queue messages MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/AMQ-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pankaj Takawale updated AMQ-5459: --------------------------------- Description: ActiveMQ Replica LevelDB cluster (3 nodes) Zookeeper 3 nodes cluster OS: CentOS 6.5 Size of messages: 50K to 500K Number of messages sent: 9,500 Consumed: around 6000 I recycled master & slave at the same time while client was sending/consuming messages. ActiveMQ cluster recovered fine, and client resumed producing/consuming messages. After around two hours, Master faced following exceptions. It demoted itself to slave. New master got elected, and it faced same issue. It kept happening. I tried to restart ActiveMQ service on all three nodes. But activemq is facing same issue. Is there any way to recover out of this state without loosing messages in queue? Nov 25 15:03:35.777 [esta] Scheduler] - ERROR AbstractStoreCursor - cbtesta - org.apache.activemq.broker.region.cursors.QueueStorePrefetch@72e4521c:testpt,batchResetNeeded=false,storeHasMessages=true,size=2516,cacheEnabled=false,maxBatchSize:200,hasSpace:true - Failed to fill batch java.lang.RuntimeException: org.apache.activemq.broker.SuppressReplyException: org.apache.activemq.broker.SuppressReplyException: ShutdownBrokerInitiated at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:280)[activemq-broker-5.10.0.jar:5.10.0] at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:162)[activemq-broker-5.10.0.jar:5.10.0] at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)[activemq-broker-5.10.0.jar:5.10.0] at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1957)[activemq-broker-5.10.0.jar:5.10.0] at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2177)[activemq-broker-5.10.0.jar:5.10.0] at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1190)[activemq-broker-5.10.0.jar:5.10.0] at org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:984)[activemq-broker-5.10.0.jar:5.10.0] at org.apache.activemq.broker.region.Queue.access$100(Queue.java:104)[activemq-broker-5.10.0.jar:5.10.0] at org.apache.activemq.broker.region.Queue$2.run(Queue.java:148)[activemq-broker-5.10.0.jar:5.10.0] at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)[activemq-client-5.10.0.jar:5.10.0] at java.util.TimerThread.mainLoop(Timer.java:555)[:1.7.0_71] at java.util.TimerThread.run(Timer.java:505)[:1.7.0_71] Nov 25 15:03:35.777 [esta] Scheduler] - ERROR Queue - cbtesta - Problem retrieving message for browse java.lang.RuntimeException: java.lang.RuntimeException: org.apache.activemq.broker.SuppressReplyException: org.apache.activemq.broker.SuppressReplyException: ShutdownBrokerInitiated at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:165)[activemq-broker-5.10.0.jar:5.10.0] at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)[activemq-broker-5.10.0.jar:5.10.0] at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1957)[activemq-broker-5.10.0.jar:5.10.0] at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2177)[activemq-broker-5.10.0.jar:5.10.0] at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1190)[activemq-broker-5.10.0.jar:5.10.0] at org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:984)[activemq-broker-5.10.0.jar:5.10.0] at org.apache.activemq.broker.region.Queue.access$100(Queue.java:104)[activemq-broker-5.10.0.jar:5.10.0] at org.apache.activemq.broker.region.Queue$2.run(Queue.java:148)[activemq-broker-5.10.0.jar:5.10.0] at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)[activemq-client-5.10.0.jar:5.10.0] at java.util.TimerThread.mainLoop(Timer.java:555)[:1.7.0_71] at java.util.TimerThread.run(Timer.java:505)[:1.7.0_71] > leveldb failed to page in queue messages > ---------------------------------------- > > Key: AMQ-5459 > URL: https://issues.apache.org/jira/browse/AMQ-5459 > Project: ActiveMQ > Issue Type: Bug > Components: Broker > Affects Versions: 5.10.0 > Environment: ActiveMQ Replica LevelDB cluster (3 nodes) > Zookeeper 3 nodes cluster > OS: CentOS 6.5 > Reporter: Pankaj Takawale > Priority: Critical > > ActiveMQ Replica LevelDB cluster (3 nodes) > Zookeeper 3 nodes cluster > OS: CentOS 6.5 > Size of messages: 50K to 500K > Number of messages sent: 9,500 > Consumed: around 6000 > I recycled master & slave at the same time while client was sending/consuming messages. > ActiveMQ cluster recovered fine, and client resumed producing/consuming messages. > After around two hours, Master faced following exceptions. It demoted itself to slave. New master got elected, and it faced same issue. It kept happening. > I tried to restart ActiveMQ service on all three nodes. But activemq is facing same issue. > Is there any way to recover out of this state without loosing messages in queue? > Nov 25 15:03:35.777 [esta] Scheduler] - ERROR AbstractStoreCursor - cbtesta - org.apache.activemq.broker.region.cursors.QueueStorePrefetch@72e4521c:testpt,batchResetNeeded=false,storeHasMessages=true,size=2516,cacheEnabled=false,maxBatchSize:200,hasSpace:true - Failed to fill batch > java.lang.RuntimeException: org.apache.activemq.broker.SuppressReplyException: org.apache.activemq.broker.SuppressReplyException: ShutdownBrokerInitiated > at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:280)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:162)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1957)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2177)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1190)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:984)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.region.Queue.access$100(Queue.java:104)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.region.Queue$2.run(Queue.java:148)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)[activemq-client-5.10.0.jar:5.10.0] > at java.util.TimerThread.mainLoop(Timer.java:555)[:1.7.0_71] > at java.util.TimerThread.run(Timer.java:505)[:1.7.0_71] > Nov 25 15:03:35.777 [esta] Scheduler] - ERROR Queue - cbtesta - Problem retrieving message for browse > java.lang.RuntimeException: java.lang.RuntimeException: org.apache.activemq.broker.SuppressReplyException: org.apache.activemq.broker.SuppressReplyException: ShutdownBrokerInitiated > at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:165)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1957)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2177)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1190)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:984)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.region.Queue.access$100(Queue.java:104)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.broker.region.Queue$2.run(Queue.java:148)[activemq-broker-5.10.0.jar:5.10.0] > at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)[activemq-client-5.10.0.jar:5.10.0] > at java.util.TimerThread.mainLoop(Timer.java:555)[:1.7.0_71] > at java.util.TimerThread.run(Timer.java:505)[:1.7.0_71] -- This message was sent by Atlassian JIRA (v6.3.4#6332)