activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <gary.tu...@gmail.com>
Subject Re: KahaDB error: "Could not locate data file"
Date Thu, 06 Jan 2011 15:41:58 GMT
AMQ-3051 is using the older AMQPersistenceAdapter and NFS, so it seems
to be different from your use case?

On 6 January 2011 15:14, dcheckoway <dcheckoway@gmail.com> wrote:
>
> Gary,
>
> Thanks for the reply and advice.  I'll definitely try disabling
> concurrentStoreAndDispatchQueues to see if that helps, and I'll set up an IO
> exception handler.
>
> Unfortunately the issue can't reliably be reproduced.  We have a cluster of
> servers all running their own local embedded queue, and it happens on one
> here, one there.
>
> That said, somebody else already opened a JIRA ticket for what looks like a
> similar/nearly identical issue:
> https://issues.apache.org/jira/browse/AMQ-3051
>
> Dan
>
>
> On Thu, Jan 6, 2011 at 9:29 AM, Gary Tully [via ActiveMQ] <
> ml-node+3177500-224584974-140184@n4.nabble.com<ml-node%2B3177500-224584974-140184@n4.nabble.com>
>> wrote:
>
>> One thought, there may be an issue with the default setting of true
>> for concurrentStoreAndDispatchQueues that is causing the index to be
>> out of sync with the store.
>>
>> It is worth trying to disable that, use:
>> <persistenceAdapter>
>>     <kahaDB directory="/usr/local/embedded/activemq-data/kahadb"
>> concurrentStoreAndDispatchQueues="false"/>
>> </persistenceAdapter>
>>
>> How often is this occurring, to really get to the bottom of this a
>> test case will be needed.
>> Does that data file eventually get written, or are there any disk space
>> issues?
>> It may help to configure an IO exception handler just in case.
>> http://activemq.apache.org/configurable-ioexception-handling.html
>>
>> Would be a good idea to create a jira issue to track this and attach
>> what ever log and application information you have to it.
>>
>> On 6 January 2011 12:39, dcheckoway <[hidden email]<http://user/SendEmail.jtp?type=node&node=3177500&i=0>>
>> wrote:
>>
>> >
>> > I'm using an embedded broker (version 5.4.2) with persistence enabled.
>>  We're
>> > pumping hundreds of millions of messages per day through this thing.
>>  Every
>> > once in a while, all of a sudden the KahaDB directory starts growing
>> > uncontrollably, and these errors start spewing out in the log over and
>> > over...
>> >
>> > --------------------------
>> > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task;
>> > AbstractStoreCursor - Failed to fill batch
>> > java.lang.RuntimeException: java.io.IOException: Could not locate data
>> file
>> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>> >        at
>> >
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
>>
>> >        at
>> >
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
>>
>> >        at
>> >
>> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
>>
>> >        at
>> org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
>> >        at
>> > org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
>> >        at
>> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
>> >        at
>> >
>> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>>
>> >        at
>> >
>> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
>> >        at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>
>> >        at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>
>> >        at java.lang.Thread.run(Thread.java:619)
>> > Caused by: java.io.IOException: Could not locate data file
>> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>> >        at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
>>
>> >        at org.apache.kahadb.journal.Journal.read(Journal.java:592)
>> >        at
>> >
>> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
>>
>> >        at
>> >
>> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
>>
>> >        at
>> >
>> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
>>
>> >        at
>> org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
>> >        at
>> >
>> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
>>
>> >        at
>> >
>> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
>>
>> >        at
>> >
>> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
>>
>> >        at
>> >
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
>>
>> >        ... 10 more
>> > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; Queue
>> -
>> > Failed to page in more queue messages
>> > java.lang.RuntimeException: java.lang.RuntimeException:
>> java.io.IOException:
>> > Could not locate data file
>> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>> >        at
>> >
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:151)
>>
>> >        at
>> >
>> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
>>
>> >        at
>> org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
>> >        at
>> > org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
>> >        at
>> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
>> >        at
>> >
>> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>>
>> >        at
>> >
>> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
>> >        at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>
>> >        at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>
>> >        at java.lang.Thread.run(Thread.java:619)
>> > Caused by: java.lang.RuntimeException: java.io.IOException: Could not
>> locate
>> > data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>>
>> >        at
>> >
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
>>
>> >        at
>> >
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
>>
>> >        ... 9 more
>> > Caused by: java.io.IOException: Could not locate data file
>> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>> >        at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
>>
>> >        at org.apache.kahadb.journal.Journal.read(Journal.java:592)
>> >        at
>> >
>> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
>>
>> >        at
>> >
>> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
>>
>> >        at
>> >
>> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
>>
>> >        at
>> org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
>> >        at
>> >
>> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
>>
>> >        at
>> >
>> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
>>
>> >        at
>> >
>> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
>>
>> >        at
>> >
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
>>
>> >        ... 10 more
>> > --------------------------
>> >
>> > It doesn't happen consistently, but it has happened numerous times.  The
>> > only way to "fix" it is to kill the app, blow away the KahaDB data
>> > directory, and restart (ugh).
>> >
>> > Has anybody else seen this issue?  Any idea why it happens?
>> >
>> > Here's the embedded broker config, fwiw:
>> > --------------------------
>> >  <broker xmlns="http://activemq.apache.org/schema/core"
>> >          id="embeddedActivemqBroker"
>> >          useJmx="false"
>> >          persistent="true"
>> >          dataDirectory="/usr/local/embedded/activemq-data"
>> >          schedulerSupport="false"
>> >          advisorySupport="false"
>> >          enableStatistics="true">
>> >    <destinationPolicy>
>> >      <policyMap>
>> >        <policyEntries>
>> >          <policyEntry queue=">" producerFlowControl="false"
>> > memoryLimit="20mb"/>
>> >        </policyEntries>
>> >      </policyMap>
>> >    </destinationPolicy>
>> >  </broker>
>> > --------------------------
>> >
>> > Help?!
>> >
>> > -----
>> > Dan Checkoway
>> > dcheckoway gmail com
>> > --
>> > View this message in context:
>> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t>
>> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >
>>
>>
>>
>> --
>> http://blog.garytully.com
>> http://fusesource.com
>>
>>
>> ------------------------------
>>  View message @
>> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177500.html
>> To unsubscribe from KahaDB error: "Could not locate data file", click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3177338&code=ZGNoZWNrb3dheUBnbWFpbC5jb218MzE3NzMzOHw3MDc4NzEwMTU=>.
>>
>>
>
>
> -----
> Dan Checkoway
> dcheckoway gmail com
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177577.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 
http://blog.garytully.com
http://fusesource.com

Mime
View raw message