Return-Path: X-Original-To: apmail-activemq-users-archive@www.apache.org Delivered-To: apmail-activemq-users-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 71EF010B19 for ; Mon, 1 Dec 2014 23:24:47 +0000 (UTC) Received: (qmail 76112 invoked by uid 500); 1 Dec 2014 23:24:47 -0000 Delivered-To: apmail-activemq-users-archive@activemq.apache.org Received: (qmail 76061 invoked by uid 500); 1 Dec 2014 23:24:47 -0000 Mailing-List: contact users-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@activemq.apache.org Delivered-To: mailing list users@activemq.apache.org Received: (qmail 76050 invoked by uid 99); 1 Dec 2014 23:24:46 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Dec 2014 23:24:46 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=SPF_SOFTFAIL,URI_HEX,URI_TRY_3LD X-Spam-Check-By: apache.org Received-SPF: softfail (nike.apache.org: transitioning domain of pankaj.takawale@gmail.com does not designate 162.253.133.43 as permitted sender) Received: from [162.253.133.43] (HELO mwork.nabble.com) (162.253.133.43) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Dec 2014 23:24:20 +0000 Received: from mjoe.nabble.com (unknown [162.253.133.57]) by mwork.nabble.com (Postfix) with ESMTP id 63CC1BD9D6E for ; Mon, 1 Dec 2014 15:24:20 -0800 (PST) Date: Mon, 1 Dec 2014 15:16:53 -0800 (PST) From: pankajtakawale To: users@activemq.apache.org Message-ID: <1417475813307-4688222.post@n4.nabble.com> Subject: Replicated LevelDB cluster: failed to page in queue messages MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Hi, I faced following exception while testing ActiveMQ leveldb cluster. On this exception, master demotes itself to slave. New master faces same exception, and it keeps on happening. Master keeps on rotating without processing messages. Client was not able to send/receive messages. I've opened jira issue: https://issues.apache.org/jira/browse/AMQ-5459 Any idea what could cause this - leveldb corrupted? Is there any way to recover out of this without loosing existing messages in queues? I would highly appreciate any pointers to fix this. Restarting activemq on all cluster nodes did not help, it ran into same issue. Nov 25 15:03:35.777 [esta] Scheduler] - ERROR AbstractStoreCursor - cbtesta - org.apache.activemq.broker.region.cursors.QueueStorePrefetch@72e4521c:testpt,batchResetNeeded=false,storeHasMessages=true,size=2516,cacheEnabled=false,maxBatchSize:200,hasSpace:true - Failed to fill batch Nov 25 15:03:35.777 [esta] Scheduler] - ERROR AbstractStoreCursor - cbtesta - org.apache.activemq.broker.region.cursors.QueueStorePrefetch@72e4521c:testpt,batchResetNeeded=false,storeHasMessages=true,size=2516,cacheEnabled=false,maxBatchSize:200,hasSpace:true - Failed to fill batch -- View this message in context: http://activemq.2283324.n4.nabble.com/Replicated-LevelDB-cluster-failed-to-page-in-queue-messages-tp4688222.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.