Return-Path: X-Original-To: apmail-activemq-dev-archive@www.apache.org Delivered-To: apmail-activemq-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 16311177FD for ; Tue, 17 Mar 2015 14:26:39 +0000 (UTC) Received: (qmail 55827 invoked by uid 500); 17 Mar 2015 14:26:38 -0000 Delivered-To: apmail-activemq-dev-archive@activemq.apache.org Received: (qmail 55762 invoked by uid 500); 17 Mar 2015 14:26:38 -0000 Mailing-List: contact dev-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@activemq.apache.org Delivered-To: mailing list dev@activemq.apache.org Received: (qmail 55750 invoked by uid 99); 17 Mar 2015 14:26:38 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Mar 2015 14:26:38 +0000 Date: Tue, 17 Mar 2015 14:26:38 +0000 (UTC) From: "Botros Kfoury (JIRA)" To: dev@activemq.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (AMQ-5670) AMQ issue MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/AMQ-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botros Kfoury updated AMQ-5670: ------------------------------- Description: Dear Support, We are having KHADB on a shared DFS storage connected to a two nodes as master slave AMQ approach. The system was running for about two weeks and suddenly it stopped as shown below in the logs, please give me your recommendations. NODE1: INFO | jvm 1 | 2015/03/04 12:54:11 | INFO | Database \\qatch-fs\AMQ_Data_PRO\data\kahadb\lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '\\qatch-fs\AMQ_Data_PRO\data\kahadb\lock' could not be locked. INFO | jvm 1 | 2015/03/04 12:54:36 | WARN | Index corrupted. Recovering the index through journal replay. Cause:java.io.IOException: The requested resource is in use INFO | jvm 1 | 2015/03/04 12:54:36 | ERROR | Failed to start Apache ActiveMQ ([CenBroker, null], java.io.IOException: Could not delete: \\qatch-fs\AMQ_Data_PRO\data\kahadb\CEN\kahadb\DLQ\queue#3a#2f#2fDLQ.#3e\db.data) INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Apache ActiveMQ 5.10.0 (CenBroker, null) is shutting down INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Connector openwire stopped INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | PListStore:[\\HQ-QTCM1\data\CenBroker\tmp_storage] stopped INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Stopping async queue tasks INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Stopping async topic tasks INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Stopped KahaDB INFO | jvm 1 | 2015/03/04 12:54:36 | ERROR | Could not stop service: KahaDB:[\\qatch-fs\AMQ_Data_PRO\data\kahadb\CEN\kahadb\DLQ\queue#3a#2f#2fDLQ.#3e]. Reason: java.lang.IllegalStateException: Cannot unload the page file when it is not loaded INFO | jvm 1 | 2015/03/04 12:54:36 | java.lang.IllegalStateException: Cannot unload the page file when it is not loaded INFO | jvm 1 | 2015/03/04 12:54:36 | at org.apache.activemq.store.kahadb.disk.page.PageFile.unload(PageFile.java:484)[activemq-kahadb-store-5.10.0.jar:5.10.0] INFO | jvm 1 | 2015/03/04 12:54:36 | at org.apache.activemq.store.kahadb.MessageDatabase.close(MessageDatabase.java:432)[activemq-kahadb-store-5.10.0.jar:5.10.0] Node2: INFO | jvm 1 | 2015/03/04 12:54:07 | ERROR | Checkpoint failed INFO | jvm 1 | 2015/03/04 12:54:07 | java.io.IOException: The requested resource is in use INFO | jvm 1 | 2015/03/04 12:54:07 | at sun.nio.ch.FileDispatcherImpl.force0(Native Method)[:1.7.0_71] INFO | jvm 1 | 2015/03/04 12:54:07 | at sun.nio.ch.FileDispatcherImpl.force(Unknown Source)[:1.7.0_71] INFO | jvm 1 | 2015/03/04 12:54:07 | at sun.nio.ch.FileChannelImpl.force(Unknown Source)[:1.7.0_71] INFO | jvm 1 | 2015/03/04 12:54:07 | at org.apache.activemq.util.RecoverableRandomAccessFile.sync(RecoverableRandomAccessFile.java:402)[activemq-kahadb-store-5.10.0.jar:5.10.0] INFO | jvm 1 | 2015/03/04 12:54:07 | at org.apache.activemq.store.kahadb.disk.page.PageFile.writeBatch(PageFile.java:1094)[activemq-kahadb-store-5.10.0.jar:5.10.0] INFO | jvm 1 | 2015/03/04 12:54:07 | at org.apache.activemq.store.kahadb.disk.page.PageFile.flush(PageFile.java:516)[activemq-kahadb-store-5.10.0.jar:5.10.0] INFO | jvm 1 | 2015/03/04 12:54:07 | at org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1512)[activemq-kahadb-store-5.10.0.jar:5.10.0] INFO | jvm 1 | 2015/03/04 12:54:07 | at org.apache.activemq.store.kahadb.MessageDatabase$17.execute(MessageDatabase.java:1484)[activemq-kahadb-store-5.10.0.jar:5.10.0] INFO | jvm 1 | 2015/03/04 12:54:07 | at org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)[activemq-kahadb-store-5.10.0.jar:5.10.0] INFO | jvm 1 | 2015/03/04 12:54:07 | at org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1481)[activemq-kahadb-store-5.10.0.jar:5.10.0] INFO | jvm 1 | 2015/03/04 12:54:07 | at org.apache.activemq.store.kahadb.MessageDatabase.checkpointCleanup(MessageDatabase.java:929)[activemq-kahadb-store-5.10.0.jar:5.10.0] INFO | jvm 1 | 2015/03/04 12:54:07 | at org.apache.activemq.store.kahadb.MessageDatabase$3.run(MessageDatabase.java:357)[activemq-kahadb-store-5.10.0.jar:5.10.0] INFO | jvm 1 | 2015/03/04 12:54:07 | INFO | Stopping BrokerService[CenBroker] due to exception, java.io.IOException: The requested resource is in use Best Regadrs, Botros Kfoury was: Dear Support, We are having KHADB connected to a two nodes as master slave AMQ approach. The system was running for about two weeks and suddenly it stopped as shown below in the logs, please give me your recommendations. NODE1: INFO | jvm 1 | 2015/03/04 12:54:11 | INFO | Database \\qatch-fs\AMQ_Data_PRO\data\kahadb\lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '\\qatch-fs\AMQ_Data_PRO\data\kahadb\lock' could not be locked. INFO | jvm 1 | 2015/03/04 12:54:36 | WARN | Index corrupted. Recovering the index through journal replay. Cause:java.io.IOException: The requested resource is in use INFO | jvm 1 | 2015/03/04 12:54:36 | ERROR | Failed to start Apache ActiveMQ ([CenBroker, null], java.io.IOException: Could not delete: \\qatch-fs\AMQ_Data_PRO\data\kahadb\CEN\kahadb\DLQ\queue#3a#2f#2fDLQ.#3e\db.data) INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Apache ActiveMQ 5.10.0 (CenBroker, null) is shutting down INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Connector openwire stopped INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | PListStore:[\\HQ-QTCM1\data\CenBroker\tmp_storage] stopped INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Stopping async queue tasks INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Stopping async topic tasks INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Stopped KahaDB INFO | jvm 1 | 2015/03/04 12:54:36 | ERROR | Could not stop service: KahaDB:[\\qatch-fs\AMQ_Data_PRO\data\kahadb\CEN\kahadb\DLQ\queue#3a#2f#2fDLQ.#3e]. Reason: java.lang.IllegalStateException: Cannot unload the page file when it is not loaded INFO | jvm 1 | 2015/03/04 12:54:36 | java.lang.IllegalStateException: Cannot unload the page file when it is not loaded INFO | jvm 1 | 2015/03/04 12:54:36 | at org.apache.activemq.store.kahadb.disk.page.PageFile.unload(PageFile.java:484)[activemq-kahadb-store-5.10.0.jar:5.10.0] INFO | jvm 1 | 2015/03/04 12:54:36 | at org.apache.activemq.store.kahadb.MessageDatabase.close(MessageDatabase.java:432)[activemq-kahadb-store-5.10.0.jar:5.10.0] Node2: INFO | jvm 1 | 2015/03/04 12:54:07 | ERROR | Checkpoint failed INFO | jvm 1 | 2015/03/04 12:54:07 | java.io.IOException: The requested resource is in use INFO | jvm 1 | 2015/03/04 12:54:07 | at sun.nio.ch.FileDispatcherImpl.force0(Native Method)[:1.7.0_71] INFO | jvm 1 | 2015/03/04 12:54:07 | at sun.nio.ch.FileDispatcherImpl.force(Unknown Source)[:1.7.0_71] INFO | jvm 1 | 2015/03/04 12:54:07 | at sun.nio.ch.FileChannelImpl.force(Unknown Source)[:1.7.0_71] INFO | jvm 1 | 2015/03/04 12:54:07 | at org.apache.activemq.util.RecoverableRandomAccessFile.sync(RecoverableRandomAccessFile.java:402)[activemq-kahadb-store-5.10.0.jar:5.10.0] INFO | jvm 1 | 2015/03/04 12:54:07 | at org.apache.activemq.store.kahadb.disk.page.PageFile.writeBatch(PageFile.java:1094)[activemq-kahadb-store-5.10.0.jar:5.10.0] INFO | jvm 1 | 2015/03/04 12:54:07 | at org.apache.activemq.store.kahadb.disk.page.PageFile.flush(PageFile.java:516)[activemq-kahadb-store-5.10.0.jar:5.10.0] INFO | jvm 1 | 2015/03/04 12:54:07 | at org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1512)[activemq-kahadb-store-5.10.0.jar:5.10.0] INFO | jvm 1 | 2015/03/04 12:54:07 | at org.apache.activemq.store.kahadb.MessageDatabase$17.execute(MessageDatabase.java:1484)[activemq-kahadb-store-5.10.0.jar:5.10.0] INFO | jvm 1 | 2015/03/04 12:54:07 | at org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)[activemq-kahadb-store-5.10.0.jar:5.10.0] INFO | jvm 1 | 2015/03/04 12:54:07 | at org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1481)[activemq-kahadb-store-5.10.0.jar:5.10.0] INFO | jvm 1 | 2015/03/04 12:54:07 | at org.apache.activemq.store.kahadb.MessageDatabase.checkpointCleanup(MessageDatabase.java:929)[activemq-kahadb-store-5.10.0.jar:5.10.0] INFO | jvm 1 | 2015/03/04 12:54:07 | at org.apache.activemq.store.kahadb.MessageDatabase$3.run(MessageDatabase.java:357)[activemq-kahadb-store-5.10.0.jar:5.10.0] INFO | jvm 1 | 2015/03/04 12:54:07 | INFO | Stopping BrokerService[CenBroker] due to exception, java.io.IOException: The requested resource is in use Best Regadrs, Botros Kfoury > AMQ issue > --------- > > Key: AMQ-5670 > URL: https://issues.apache.org/jira/browse/AMQ-5670 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB > Affects Versions: 5.10.0 > Reporter: Botros Kfoury > Fix For: 5.x > > > Dear Support, > We are having KHADB on a shared DFS storage connected to a two nodes as master slave AMQ approach. The system was running for about two weeks and suddenly it stopped as shown below in the logs, please give me your recommendations. > NODE1: > INFO | jvm 1 | 2015/03/04 12:54:11 | INFO | Database \\qatch-fs\AMQ_Data_PRO\data\kahadb\lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '\\qatch-fs\AMQ_Data_PRO\data\kahadb\lock' could not be locked. > INFO | jvm 1 | 2015/03/04 12:54:36 | WARN | Index corrupted. Recovering the index through journal replay. Cause:java.io.IOException: The requested resource is in use > INFO | jvm 1 | 2015/03/04 12:54:36 | ERROR | Failed to start Apache ActiveMQ ([CenBroker, null], java.io.IOException: Could not delete: \\qatch-fs\AMQ_Data_PRO\data\kahadb\CEN\kahadb\DLQ\queue#3a#2f#2fDLQ.#3e\db.data) > INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Apache ActiveMQ 5.10.0 (CenBroker, null) is shutting down > INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Connector openwire stopped > INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | PListStore:[\\HQ-QTCM1\data\CenBroker\tmp_storage] stopped > INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Stopping async queue tasks > INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Stopping async topic tasks > INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Stopped KahaDB > INFO | jvm 1 | 2015/03/04 12:54:36 | ERROR | Could not stop service: KahaDB:[\\qatch-fs\AMQ_Data_PRO\data\kahadb\CEN\kahadb\DLQ\queue#3a#2f#2fDLQ.#3e]. Reason: java.lang.IllegalStateException: Cannot unload the page file when it is not loaded > INFO | jvm 1 | 2015/03/04 12:54:36 | java.lang.IllegalStateException: Cannot unload the page file when it is not loaded > INFO | jvm 1 | 2015/03/04 12:54:36 | > at org.apache.activemq.store.kahadb.disk.page.PageFile.unload(PageFile.java:484)[activemq-kahadb-store-5.10.0.jar:5.10.0] > INFO | jvm 1 | 2015/03/04 12:54:36 | > at org.apache.activemq.store.kahadb.MessageDatabase.close(MessageDatabase.java:432)[activemq-kahadb-store-5.10.0.jar:5.10.0] > Node2: > INFO | jvm 1 | 2015/03/04 12:54:07 | ERROR | Checkpoint failed > INFO | jvm 1 | 2015/03/04 12:54:07 | java.io.IOException: The requested resource is in use > INFO | jvm 1 | 2015/03/04 12:54:07 | > at sun.nio.ch.FileDispatcherImpl.force0(Native Method)[:1.7.0_71] > INFO | jvm 1 | 2015/03/04 12:54:07 | > at sun.nio.ch.FileDispatcherImpl.force(Unknown Source)[:1.7.0_71] > INFO | jvm 1 | 2015/03/04 12:54:07 | > at sun.nio.ch.FileChannelImpl.force(Unknown Source)[:1.7.0_71] > INFO | jvm 1 | 2015/03/04 12:54:07 | > at org.apache.activemq.util.RecoverableRandomAccessFile.sync(RecoverableRandomAccessFile.java:402)[activemq-kahadb-store-5.10.0.jar:5.10.0] > INFO | jvm 1 | 2015/03/04 12:54:07 | > at org.apache.activemq.store.kahadb.disk.page.PageFile.writeBatch(PageFile.java:1094)[activemq-kahadb-store-5.10.0.jar:5.10.0] > INFO | jvm 1 | 2015/03/04 12:54:07 | > at org.apache.activemq.store.kahadb.disk.page.PageFile.flush(PageFile.java:516)[activemq-kahadb-store-5.10.0.jar:5.10.0] > INFO | jvm 1 | 2015/03/04 12:54:07 | > at org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1512)[activemq-kahadb-store-5.10.0.jar:5.10.0] > INFO | jvm 1 | 2015/03/04 12:54:07 | > at org.apache.activemq.store.kahadb.MessageDatabase$17.execute(MessageDatabase.java:1484)[activemq-kahadb-store-5.10.0.jar:5.10.0] > INFO | jvm 1 | 2015/03/04 12:54:07 | > at org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)[activemq-kahadb-store-5.10.0.jar:5.10.0] > INFO | jvm 1 | 2015/03/04 12:54:07 | > at org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1481)[activemq-kahadb-store-5.10.0.jar:5.10.0] > INFO | jvm 1 | 2015/03/04 12:54:07 | > at org.apache.activemq.store.kahadb.MessageDatabase.checkpointCleanup(MessageDatabase.java:929)[activemq-kahadb-store-5.10.0.jar:5.10.0] > INFO | jvm 1 | 2015/03/04 12:54:07 | > at org.apache.activemq.store.kahadb.MessageDatabase$3.run(MessageDatabase.java:357)[activemq-kahadb-store-5.10.0.jar:5.10.0] > INFO | jvm 1 | 2015/03/04 12:54:07 | INFO | Stopping BrokerService[CenBroker] due to exception, java.io.IOException: The requested resource is in use > Best Regadrs, > Botros Kfoury -- This message was sent by Atlassian JIRA (v6.3.4#6332)