activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Botros Kfoury (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AMQ-5670) AMQ issue
Date Tue, 17 Mar 2015 14:26:38 GMT

     [ https://issues.apache.org/jira/browse/AMQ-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Botros Kfoury updated AMQ-5670:
-------------------------------
    Description: 
Dear Support,

We are having KHADB on a shared DFS storage connected to a two nodes as master slave AMQ approach.
The system was running for about two weeks and suddenly it stopped as shown below in the logs,
please give me your recommendations.

NODE1:
INFO   | jvm 1    | 2015/03/04 12:54:11 |  INFO | Database \\qatch-fs\AMQ_Data_PRO\data\kahadb\lock
is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException:
File '\\qatch-fs\AMQ_Data_PRO\data\kahadb\lock' could not be locked.
INFO   | jvm 1    | 2015/03/04 12:54:36 |  WARN | Index corrupted. Recovering the index through
journal replay. Cause:java.io.IOException: The requested resource is in use
INFO   | jvm 1    | 2015/03/04 12:54:36 | ERROR | Failed to start Apache ActiveMQ ([CenBroker,
null], java.io.IOException: Could not delete: \\qatch-fs\AMQ_Data_PRO\data\kahadb\CEN\kahadb\DLQ\queue#3a#2f#2fDLQ.#3e\db.data)
INFO   | jvm 1    | 2015/03/04 12:54:36 |  INFO | Apache ActiveMQ 5.10.0 (CenBroker, null)
is shutting down
INFO   | jvm 1    | 2015/03/04 12:54:36 |  INFO | Connector openwire stopped
INFO   | jvm 1    | 2015/03/04 12:54:36 |  INFO | PListStore:[\\HQ-QTCM1\data\CenBroker\tmp_storage]
stopped
INFO   | jvm 1    | 2015/03/04 12:54:36 |  INFO | Stopping async queue tasks
INFO   | jvm 1    | 2015/03/04 12:54:36 |  INFO | Stopping async topic tasks
INFO   | jvm 1    | 2015/03/04 12:54:36 |  INFO | Stopped KahaDB
INFO   | jvm 1    | 2015/03/04 12:54:36 | ERROR | Could not stop service: KahaDB:[\\qatch-fs\AMQ_Data_PRO\data\kahadb\CEN\kahadb\DLQ\queue#3a#2f#2fDLQ.#3e].
Reason: java.lang.IllegalStateException: Cannot unload the page file when it is not loaded
INFO   | jvm 1    | 2015/03/04 12:54:36 | java.lang.IllegalStateException: Cannot unload the
page file when it is not loaded
INFO   | jvm 1    | 2015/03/04 12:54:36 | 
at org.apache.activemq.store.kahadb.disk.page.PageFile.unload(PageFile.java:484)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO   | jvm 1    | 2015/03/04 12:54:36 | 
at org.apache.activemq.store.kahadb.MessageDatabase.close(MessageDatabase.java:432)[activemq-kahadb-store-5.10.0.jar:5.10.0]

Node2:
INFO   | jvm 1    | 2015/03/04 12:54:07 | ERROR | Checkpoint failed
INFO   | jvm 1    | 2015/03/04 12:54:07 | java.io.IOException: The requested resource is in
use
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at sun.nio.ch.FileDispatcherImpl.force0(Native Method)[:1.7.0_71]
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at sun.nio.ch.FileDispatcherImpl.force(Unknown Source)[:1.7.0_71]
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at sun.nio.ch.FileChannelImpl.force(Unknown Source)[:1.7.0_71]
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at org.apache.activemq.util.RecoverableRandomAccessFile.sync(RecoverableRandomAccessFile.java:402)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at org.apache.activemq.store.kahadb.disk.page.PageFile.writeBatch(PageFile.java:1094)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at org.apache.activemq.store.kahadb.disk.page.PageFile.flush(PageFile.java:516)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1512)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at org.apache.activemq.store.kahadb.MessageDatabase$17.execute(MessageDatabase.java:1484)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1481)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at org.apache.activemq.store.kahadb.MessageDatabase.checkpointCleanup(MessageDatabase.java:929)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at org.apache.activemq.store.kahadb.MessageDatabase$3.run(MessageDatabase.java:357)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO   | jvm 1    | 2015/03/04 12:54:07 |  INFO | Stopping BrokerService[CenBroker] due to
exception, java.io.IOException: The requested resource is in use


Best Regadrs,
Botros Kfoury



  was:
Dear Support,

We are having KHADB connected to a two nodes as master slave AMQ approach. The system was
running for about two weeks and suddenly it stopped as shown below in the logs, please give
me your recommendations.

NODE1:
INFO   | jvm 1    | 2015/03/04 12:54:11 |  INFO | Database \\qatch-fs\AMQ_Data_PRO\data\kahadb\lock
is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException:
File '\\qatch-fs\AMQ_Data_PRO\data\kahadb\lock' could not be locked.
INFO   | jvm 1    | 2015/03/04 12:54:36 |  WARN | Index corrupted. Recovering the index through
journal replay. Cause:java.io.IOException: The requested resource is in use
INFO   | jvm 1    | 2015/03/04 12:54:36 | ERROR | Failed to start Apache ActiveMQ ([CenBroker,
null], java.io.IOException: Could not delete: \\qatch-fs\AMQ_Data_PRO\data\kahadb\CEN\kahadb\DLQ\queue#3a#2f#2fDLQ.#3e\db.data)
INFO   | jvm 1    | 2015/03/04 12:54:36 |  INFO | Apache ActiveMQ 5.10.0 (CenBroker, null)
is shutting down
INFO   | jvm 1    | 2015/03/04 12:54:36 |  INFO | Connector openwire stopped
INFO   | jvm 1    | 2015/03/04 12:54:36 |  INFO | PListStore:[\\HQ-QTCM1\data\CenBroker\tmp_storage]
stopped
INFO   | jvm 1    | 2015/03/04 12:54:36 |  INFO | Stopping async queue tasks
INFO   | jvm 1    | 2015/03/04 12:54:36 |  INFO | Stopping async topic tasks
INFO   | jvm 1    | 2015/03/04 12:54:36 |  INFO | Stopped KahaDB
INFO   | jvm 1    | 2015/03/04 12:54:36 | ERROR | Could not stop service: KahaDB:[\\qatch-fs\AMQ_Data_PRO\data\kahadb\CEN\kahadb\DLQ\queue#3a#2f#2fDLQ.#3e].
Reason: java.lang.IllegalStateException: Cannot unload the page file when it is not loaded
INFO   | jvm 1    | 2015/03/04 12:54:36 | java.lang.IllegalStateException: Cannot unload the
page file when it is not loaded
INFO   | jvm 1    | 2015/03/04 12:54:36 | 
at org.apache.activemq.store.kahadb.disk.page.PageFile.unload(PageFile.java:484)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO   | jvm 1    | 2015/03/04 12:54:36 | 
at org.apache.activemq.store.kahadb.MessageDatabase.close(MessageDatabase.java:432)[activemq-kahadb-store-5.10.0.jar:5.10.0]

Node2:
INFO   | jvm 1    | 2015/03/04 12:54:07 | ERROR | Checkpoint failed
INFO   | jvm 1    | 2015/03/04 12:54:07 | java.io.IOException: The requested resource is in
use
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at sun.nio.ch.FileDispatcherImpl.force0(Native Method)[:1.7.0_71]
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at sun.nio.ch.FileDispatcherImpl.force(Unknown Source)[:1.7.0_71]
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at sun.nio.ch.FileChannelImpl.force(Unknown Source)[:1.7.0_71]
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at org.apache.activemq.util.RecoverableRandomAccessFile.sync(RecoverableRandomAccessFile.java:402)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at org.apache.activemq.store.kahadb.disk.page.PageFile.writeBatch(PageFile.java:1094)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at org.apache.activemq.store.kahadb.disk.page.PageFile.flush(PageFile.java:516)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1512)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at org.apache.activemq.store.kahadb.MessageDatabase$17.execute(MessageDatabase.java:1484)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1481)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at org.apache.activemq.store.kahadb.MessageDatabase.checkpointCleanup(MessageDatabase.java:929)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO   | jvm 1    | 2015/03/04 12:54:07 | 
at org.apache.activemq.store.kahadb.MessageDatabase$3.run(MessageDatabase.java:357)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO   | jvm 1    | 2015/03/04 12:54:07 |  INFO | Stopping BrokerService[CenBroker] due to
exception, java.io.IOException: The requested resource is in use


Best Regadrs,
Botros Kfoury




> AMQ issue
> ---------
>
>                 Key: AMQ-5670
>                 URL: https://issues.apache.org/jira/browse/AMQ-5670
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: KahaDB
>    Affects Versions: 5.10.0
>            Reporter: Botros Kfoury
>             Fix For: 5.x
>
>
> Dear Support,
> We are having KHADB on a shared DFS storage connected to a two nodes as master slave
AMQ approach. The system was running for about two weeks and suddenly it stopped as shown
below in the logs, please give me your recommendations.
> NODE1:
> INFO   | jvm 1    | 2015/03/04 12:54:11 |  INFO | Database \\qatch-fs\AMQ_Data_PRO\data\kahadb\lock
is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException:
File '\\qatch-fs\AMQ_Data_PRO\data\kahadb\lock' could not be locked.
> INFO   | jvm 1    | 2015/03/04 12:54:36 |  WARN | Index corrupted. Recovering the index
through journal replay. Cause:java.io.IOException: The requested resource is in use
> INFO   | jvm 1    | 2015/03/04 12:54:36 | ERROR | Failed to start Apache ActiveMQ ([CenBroker,
null], java.io.IOException: Could not delete: \\qatch-fs\AMQ_Data_PRO\data\kahadb\CEN\kahadb\DLQ\queue#3a#2f#2fDLQ.#3e\db.data)
> INFO   | jvm 1    | 2015/03/04 12:54:36 |  INFO | Apache ActiveMQ 5.10.0 (CenBroker,
null) is shutting down
> INFO   | jvm 1    | 2015/03/04 12:54:36 |  INFO | Connector openwire stopped
> INFO   | jvm 1    | 2015/03/04 12:54:36 |  INFO | PListStore:[\\HQ-QTCM1\data\CenBroker\tmp_storage]
stopped
> INFO   | jvm 1    | 2015/03/04 12:54:36 |  INFO | Stopping async queue tasks
> INFO   | jvm 1    | 2015/03/04 12:54:36 |  INFO | Stopping async topic tasks
> INFO   | jvm 1    | 2015/03/04 12:54:36 |  INFO | Stopped KahaDB
> INFO   | jvm 1    | 2015/03/04 12:54:36 | ERROR | Could not stop service: KahaDB:[\\qatch-fs\AMQ_Data_PRO\data\kahadb\CEN\kahadb\DLQ\queue#3a#2f#2fDLQ.#3e].
Reason: java.lang.IllegalStateException: Cannot unload the page file when it is not loaded
> INFO   | jvm 1    | 2015/03/04 12:54:36 | java.lang.IllegalStateException: Cannot unload
the page file when it is not loaded
> INFO   | jvm 1    | 2015/03/04 12:54:36 | 
> at org.apache.activemq.store.kahadb.disk.page.PageFile.unload(PageFile.java:484)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> INFO   | jvm 1    | 2015/03/04 12:54:36 | 
> at org.apache.activemq.store.kahadb.MessageDatabase.close(MessageDatabase.java:432)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> Node2:
> INFO   | jvm 1    | 2015/03/04 12:54:07 | ERROR | Checkpoint failed
> INFO   | jvm 1    | 2015/03/04 12:54:07 | java.io.IOException: The requested resource
is in use
> INFO   | jvm 1    | 2015/03/04 12:54:07 | 
> at sun.nio.ch.FileDispatcherImpl.force0(Native Method)[:1.7.0_71]
> INFO   | jvm 1    | 2015/03/04 12:54:07 | 
> at sun.nio.ch.FileDispatcherImpl.force(Unknown Source)[:1.7.0_71]
> INFO   | jvm 1    | 2015/03/04 12:54:07 | 
> at sun.nio.ch.FileChannelImpl.force(Unknown Source)[:1.7.0_71]
> INFO   | jvm 1    | 2015/03/04 12:54:07 | 
> at org.apache.activemq.util.RecoverableRandomAccessFile.sync(RecoverableRandomAccessFile.java:402)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> INFO   | jvm 1    | 2015/03/04 12:54:07 | 
> at org.apache.activemq.store.kahadb.disk.page.PageFile.writeBatch(PageFile.java:1094)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> INFO   | jvm 1    | 2015/03/04 12:54:07 | 
> at org.apache.activemq.store.kahadb.disk.page.PageFile.flush(PageFile.java:516)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> INFO   | jvm 1    | 2015/03/04 12:54:07 | 
> at org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1512)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> INFO   | jvm 1    | 2015/03/04 12:54:07 | 
> at org.apache.activemq.store.kahadb.MessageDatabase$17.execute(MessageDatabase.java:1484)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> INFO   | jvm 1    | 2015/03/04 12:54:07 | 
> at org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> INFO   | jvm 1    | 2015/03/04 12:54:07 | 
> at org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1481)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> INFO   | jvm 1    | 2015/03/04 12:54:07 | 
> at org.apache.activemq.store.kahadb.MessageDatabase.checkpointCleanup(MessageDatabase.java:929)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> INFO   | jvm 1    | 2015/03/04 12:54:07 | 
> at org.apache.activemq.store.kahadb.MessageDatabase$3.run(MessageDatabase.java:357)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> INFO   | jvm 1    | 2015/03/04 12:54:07 |  INFO | Stopping BrokerService[CenBroker] due
to exception, java.io.IOException: The requested resource is in use
> Best Regadrs,
> Botros Kfoury



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message