activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Checkoway (JIRA)" <j...@apache.org>
Subject [jira] Created: (AMQ-3120) KahaDB error: "Could not locate data file"
Date Thu, 06 Jan 2011 16:48:45 GMT
KahaDB error: "Could not locate data file"
------------------------------------------

                 Key: AMQ-3120
                 URL: https://issues.apache.org/jira/browse/AMQ-3120
             Project: ActiveMQ
          Issue Type: Bug
          Components: Broker
    Affects Versions: 5.4.2
         Environment: linux: CentOS 5.5
happens on both dual- and quad-cpu boxes, 4gb ram

java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
            Reporter: Dan Checkoway


I'm using an embedded broker (version 5.4.2) with persistence enabled.  We're pumping hundreds
of millions of messages per day through this thing.  Every once in a while, all of a sudden
the KahaDB directory starts growing uncontrollably, and these errors start spewing out in
the log over and over...

--------------------------
ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; AbstractStoreCursor - Failed
to fill batch
java.lang.RuntimeException: java.io.IOException: Could not locate data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
        at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
        at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
        at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
        at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
        at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
        at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Could not locate data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
        at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
        at org.apache.kahadb.journal.Journal.read(Journal.java:592)
        at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
        at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
        at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
        at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
        at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
        at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
        at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
        ... 10 more
ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; Queue - Failed to page
in more queue messages
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate
data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:151)
        at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
        at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
        at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
        at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
        at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
        at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
        ... 9 more
Caused by: java.io.IOException: Could not locate data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
        at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
        at org.apache.kahadb.journal.Journal.read(Journal.java:592)
        at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
        at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
        at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
        at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
        at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
        at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
        at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
        ... 10 more
--------------------------

It doesn't happen consistently, but it has happened numerous times.  The only way to "fix"
it is to kill the app, blow away the KahaDB data directory, and restart (ugh).

Unfortunately this issue is not reproduceable.

Here's the embedded broker config, fwiw:
--------------------------
  <broker xmlns="http://activemq.apache.org/schema/core"
          id="embeddedActivemqBroker"
          useJmx="false"
          persistent="true"
          dataDirectory="/usr/local/embedded/activemq-data"
          schedulerSupport="false"
          advisorySupport="false"
          enableStatistics="true">
    <destinationPolicy>
      <policyMap>
        <policyEntries>
          <policyEntry queue=">" producerFlowControl="false" memoryLimit="20mb"/>
        </policyEntries>
      </policyMap>
    </destinationPolicy>
  </broker>
-------------------------- 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message