activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Checkoway (JIRA)" <j...@apache.org>
Subject [jira] Commented: (AMQ-3120) KahaDB error: "Could not locate data file"
Date Fri, 21 Jan 2011 20:21:44 GMT

    [ https://issues.apache.org/jira/browse/AMQ-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12984911#action_12984911
] 

Dan Checkoway commented on AMQ-3120:
------------------------------------

@Gary, AMQ-3122 talks about broker running out of disk space being a cause.  Can you clarify
if that means actually running out of physical space on the disk, or hitting a systemUsage
limit for storeUsage or something like that?  I can tell you definitively that our setup did
NOT run out of physical disk space.  Whether it hit the storeUsage limit or not (it wasn't
explicitly configured, so whatever is the default), that's another story...

Thanks for following up on this!

> KahaDB error: "Could not locate data file"
> ------------------------------------------
>
>                 Key: AMQ-3120
>                 URL: https://issues.apache.org/jira/browse/AMQ-3120
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.4.2
>         Environment: linux: CentOS 5.5
> happens on both dual- and quad-cpu boxes, 4gb ram
> java version "1.6.0_21"
> Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
> Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
>            Reporter: Dan Checkoway
>
> I'm using an embedded broker (version 5.4.2) with persistence enabled.  We're pumping
hundreds of millions of messages per day through this thing.  Every once in a while, all of
a sudden the KahaDB directory starts growing uncontrollably, and these errors start spewing
out in the log over and over...
> --------------------------
> ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; AbstractStoreCursor
- Failed to fill batch
> java.lang.RuntimeException: java.io.IOException: Could not locate data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>         at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
>         at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
>         at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
>         at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
>         at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
>         at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
>         at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>         at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: Could not locate data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>         at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
>         at org.apache.kahadb.journal.Journal.read(Journal.java:592)
>         at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
>         at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
>         at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
>         at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
>         at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
>         at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
>         at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
>         at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
>         ... 10 more
> ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; Queue - Failed to
page in more queue messages
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not
locate data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>         at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:151)
>         at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
>         at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
>         at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
>         at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
>         at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>         at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file
/usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>         at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
>         at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
>         ... 9 more
> Caused by: java.io.IOException: Could not locate data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>         at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
>         at org.apache.kahadb.journal.Journal.read(Journal.java:592)
>         at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
>         at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
>         at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
>         at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
>         at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
>         at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
>         at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
>         at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
>         ... 10 more
> --------------------------
> It doesn't happen consistently, but it has happened numerous times.  The only way to
"fix" it is to kill the app, blow away the KahaDB data directory, and restart (ugh).
> Unfortunately this issue is not reproduceable.
> Here's the embedded broker config, fwiw:
> --------------------------
>   <broker xmlns="http://activemq.apache.org/schema/core"
>           id="embeddedActivemqBroker"
>           useJmx="false"
>           persistent="true"
>           dataDirectory="/usr/local/embedded/activemq-data"
>           schedulerSupport="false"
>           advisorySupport="false"
>           enableStatistics="true">
>     <destinationPolicy>
>       <policyMap>
>         <policyEntries>
>           <policyEntry queue=">" producerFlowControl="false" memoryLimit="20mb"/>
>         </policyEntries>
>       </policyMap>
>     </destinationPolicy>
>   </broker>
> -------------------------- 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message