Return-Path: X-Original-To: apmail-activemq-dev-archive@www.apache.org Delivered-To: apmail-activemq-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2C609D404 for ; Sat, 15 Sep 2012 00:21:08 +0000 (UTC) Received: (qmail 8565 invoked by uid 500); 15 Sep 2012 00:21:07 -0000 Delivered-To: apmail-activemq-dev-archive@activemq.apache.org Received: (qmail 8495 invoked by uid 500); 15 Sep 2012 00:21:07 -0000 Mailing-List: contact dev-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@activemq.apache.org Delivered-To: mailing list dev@activemq.apache.org Received: (qmail 8260 invoked by uid 99); 15 Sep 2012 00:21:07 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Sep 2012 00:21:07 +0000 Date: Sat, 15 Sep 2012 11:21:07 +1100 (NCT) From: "Edwin Yu (JIRA)" To: dev@activemq.apache.org Message-ID: <1562479061.83244.1347668467748.JavaMail.jiratomcat@arcas> In-Reply-To: <701140707.83200.1347667507633.JavaMail.jiratomcat@arcas> Subject: [jira] [Updated] (AMQ-4051) IOException: Could not locate data file .\db-10.log MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/AMQ-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwin Yu updated AMQ-4051: -------------------------- Summary: IOException: Could not locate data file .\db-10.log (was: IOException: Could not locate data file .\esb\db-10.log) > IOException: Could not locate data file .\db-10.log > --------------------------------------------------- > > Key: AMQ-4051 > URL: https://issues.apache.org/jira/browse/AMQ-4051 > Project: ActiveMQ > Issue Type: Bug > Components: Broker > Affects Versions: 5.6.0 > Environment: Windows Server 2008 R2, 8GB, Intel Core 2 Duo CPU 64 bit > JDK 1.6.0_26, 64 bit > Reporter: Edwin Yu > Priority: Critical > Fix For: 5.7.0 > > Attachments: activemq.xml > > > We're doing load testing on the ActiveMQ 5.6.0 broker-to-broker connection, pumping in about 80,000 messages per hour into the queue. One of the brokers stops dequeueing messages to the consumer while its queue size increases. The log contains repeating errors about "could not locate the data file." > We encountered this same issue couple weeks ago. I found AMQ-3120 and added ignoreMissingJouralFiles="true" like that issue suggested. After many more days of testing, today we countered this same problem again. Our disk space is plenty. The log is listed below. I'm attached our activemq configuration from the troubled broker to this issue. > ------------------------------------------------------------------ > 2012-09-14 16:01:47,354 ERROR | BrokerService[xps-amq-broker_HPC-HAIFA-HAIFA-HAIFA] Task-18 | [Journal:352] Looking for key 10 but not found in fileMap: {11=db-11.log number = 11 , length = 499163} > 2012-09-14 16:01:47,354 ERROR | BrokerService[xps-amq-broker_HPC-HAIFA-HAIFA-HAIFA] Task-18 | [AbstractStoreCursor:279] org.apache.activemq.broker.region.cursors.QueueStorePrefetch@5fae7502:n4.HPC/HAIFA/HAIFA/HAIFA,batchResetNeeded=false,storeHasMessages=true,size=100,cacheEnabled=false - Failed to fill batch > java.io.IOException: Could not locate data file .\esb\amq_N4-PERFORM70\db-10.log > at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:353) > at org.apache.kahadb.journal.Journal.read(Journal.java:600) > at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:924) > at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:1015) > at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:556) > at org.apache.kahadb.page.Transaction.execute(Transaction.java:769) > at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:545) > at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:106) > at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97) > at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:277) > at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113) > at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157) > at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1766) > at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1995) > at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1488) > at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) > at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > 2012-09-14 16:01:47,354 ERROR | BrokerService[xps-amq-broker_HPC-HAIFA-HAIFA-HAIFA] Task-18 | [AbstractStoreCursor:115] org.apache.activemq.broker.region.cursors.QueueStorePrefetch@5fae7502:n4.HPC/HAIFA/HAIFA/HAIFA,batchResetNeeded=false,storeHasMessages=true,size=100,cacheEnabled=false - Failed to fill batch > java.lang.RuntimeException: java.io.IOException: Could not locate data file .\esb\amq_N4-PERFORM70\db-10.log > at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:280) > at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113) > at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157) > at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1766) > at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1995) > at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1488) > at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) > at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.IOException: Could not locate data file .\esb\amq_N4-PERFORM70\db-10.log > at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:353) > at org.apache.kahadb.journal.Journal.read(Journal.java:600) > at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:924) > at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:1015) > at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:556) > at org.apache.kahadb.page.Transaction.execute(Transaction.java:769) > at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:545) > at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:106) > at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97) > at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:277) > ... 10 more > 2012-09-14 16:01:47,354 ERROR | BrokerService[xps-amq-broker_HPC-HAIFA-HAIFA-HAIFA] Task-18 | [Queue:1491] Failed to page in more queue messages > java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file .\esb\amq_N4-PERFORM70\db-10.log > at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:116) > at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157) > at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1766) > at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1995) > at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1488) > at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) > at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file .\esb\amq_N4-PERFORM70\db-10.log > at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:280) > at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113) > ... 9 more > Caused by: java.io.IOException: Could not locate data file .\esb\amq_N4-PERFORM70\db-10.log > at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:353) > at org.apache.kahadb.journal.Journal.read(Journal.java:600) > at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:924) > at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:1015) > at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:556) > at org.apache.kahadb.page.Transaction.execute(Transaction.java:769) > at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:545) > at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:106) > at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97) > at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:277) > ... 10 more > 2012-09-14 16:01:47,354 INFO | ActiveMQ Session Task-26 | [LogComponent:41] > -------------------------------------------------------------------------- -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira