activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gary Tully (JIRA)" <j...@apache.org>
Subject [jira] Reopened: (AMQ-3120) KahaDB error: "Could not locate data file"
Date Mon, 14 Feb 2011 13:17:57 GMT

     [ https://issues.apache.org/jira/browse/AMQ-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Gary Tully reopened AMQ-3120:
-----------------------------


Unit test (org.apache.activemq.bugs.AMQ3120Test) shows intermittent failure, so the problem
still exists.

> KahaDB error: "Could not locate data file"
> ------------------------------------------
>
>                 Key: AMQ-3120
>                 URL: https://issues.apache.org/jira/browse/AMQ-3120
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.4.2
>         Environment: linux: CentOS 5.5
> happens on both dual- and quad-cpu boxes, 4gb ram
> java version "1.6.0_21"
> Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
> Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
>            Reporter: Dan Checkoway
>            Assignee: Gary Tully
>             Fix For: 5.5.0
>
>
> I'm using an embedded broker (version 5.4.2) with persistence enabled.  We're pumping
hundreds of millions of messages per day through this thing.  Every once in a while, all of
a sudden the KahaDB directory starts growing uncontrollably, and these errors start spewing
out in the log over and over...
> --------------------------
> ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; AbstractStoreCursor
- Failed to fill batch
> java.lang.RuntimeException: java.io.IOException: Could not locate data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>         at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
>         at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
>         at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
>         at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
>         at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
>         at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
>         at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>         at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: Could not locate data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>         at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
>         at org.apache.kahadb.journal.Journal.read(Journal.java:592)
>         at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
>         at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
>         at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
>         at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
>         at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
>         at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
>         at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
>         at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
>         ... 10 more
> ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; Queue - Failed to
page in more queue messages
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not
locate data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>         at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:151)
>         at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
>         at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
>         at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
>         at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
>         at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>         at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file
/usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>         at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
>         at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
>         ... 9 more
> Caused by: java.io.IOException: Could not locate data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>         at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
>         at org.apache.kahadb.journal.Journal.read(Journal.java:592)
>         at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
>         at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
>         at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
>         at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
>         at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
>         at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
>         at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
>         at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
>         ... 10 more
> --------------------------
> It doesn't happen consistently, but it has happened numerous times.  The only way to
"fix" it is to kill the app, blow away the KahaDB data directory, and restart (ugh).
> Unfortunately this issue is not reproduceable.
> Here's the embedded broker config, fwiw:
> --------------------------
>   <broker xmlns="http://activemq.apache.org/schema/core"
>           id="embeddedActivemqBroker"
>           useJmx="false"
>           persistent="true"
>           dataDirectory="/usr/local/embedded/activemq-data"
>           schedulerSupport="false"
>           advisorySupport="false"
>           enableStatistics="true">
>     <destinationPolicy>
>       <policyMap>
>         <policyEntries>
>           <policyEntry queue=">" producerFlowControl="false" memoryLimit="20mb"/>
>         </policyEntries>
>       </policyMap>
>     </destinationPolicy>
>   </broker>
> -------------------------- 

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message