activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dcheckoway <dchecko...@gmail.com>
Subject KahaDB error: "Could not locate data file"
Date Thu, 06 Jan 2011 12:39:35 GMT

I'm using an embedded broker (version 5.4.2) with persistence enabled.  We're
pumping hundreds of millions of messages per day through this thing.  Every
once in a while, all of a sudden the KahaDB directory starts growing
uncontrollably, and these errors start spewing out in the log over and
over...

--------------------------
ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task;
AbstractStoreCursor - Failed to fill batch
java.lang.RuntimeException: java.io.IOException: Could not locate data file
/usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
        at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
        at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
        at
org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
        at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
        at
org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
        at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
        at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
        at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Could not locate data file
/usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
        at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
        at org.apache.kahadb.journal.Journal.read(Journal.java:592)
        at
org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
        at
org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
        at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
        at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
        at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
        at
org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
        at
org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
        at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
        ... 10 more
ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; Queue -
Failed to page in more queue messages 
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException:
Could not locate data file
/usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
        at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:151)
        at
org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
        at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
        at
org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
        at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
        at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
        at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate
data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
        at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
        at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
        ... 9 more
Caused by: java.io.IOException: Could not locate data file
/usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
        at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
        at org.apache.kahadb.journal.Journal.read(Journal.java:592)
        at
org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
        at
org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
        at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
        at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
        at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
        at
org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
        at
org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
        at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
        ... 10 more
--------------------------

It doesn't happen consistently, but it has happened numerous times.  The
only way to "fix" it is to kill the app, blow away the KahaDB data
directory, and restart (ugh).

Has anybody else seen this issue?  Any idea why it happens?

Here's the embedded broker config, fwiw:
--------------------------
  <broker xmlns="http://activemq.apache.org/schema/core"
          id="embeddedActivemqBroker"
          useJmx="false"
          persistent="true"
          dataDirectory="/usr/local/embedded/activemq-data"
          schedulerSupport="false"
          advisorySupport="false"
          enableStatistics="true">
    <destinationPolicy>
      <policyMap>
        <policyEntries>
          <policyEntry queue=">" producerFlowControl="false"
memoryLimit="20mb"/>
        </policyEntries>
      </policyMap>
    </destinationPolicy>
  </broker>
--------------------------

Help?!

-----
Dan Checkoway
dcheckoway gmail com
-- 
View this message in context: http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message