activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Norman Maurer <nor...@apache.org>
Subject Re: KahaDB error: "Could not locate data file"
Date Thu, 06 Jan 2011 16:48:41 GMT
In 5.4.2 its kahadb..

Bye,
Norman

2011/1/6 dcheckoway <dcheckoway@gmail.com>:
>
> Yeah, we're using whatever is the default persistence provider in 5.4.2.  I
> opened a fresh ticket for it:
>
> https://issues.apache.org/jira/browse/AMQ-3120
>
> Thanks,
> Dan
>
> On Thu, Jan 6, 2011 at 10:42 AM, Gary Tully [via ActiveMQ] <
> ml-node+3177622-1458695149-140184@n4.nabble.com<ml-node%2B3177622-1458695149-140184@n4.nabble.com>
>> wrote:
>
>> AMQ-3051 is using the older AMQPersistenceAdapter and NFS, so it seems
>> to be different from your use case?
>>
>> On 6 January 2011 15:14, dcheckoway <[hidden email]<http://user/SendEmail.jtp?type=node&node=3177622&i=0>>
>> wrote:
>>
>> >
>> > Gary,
>> >
>> > Thanks for the reply and advice.  I'll definitely try disabling
>> > concurrentStoreAndDispatchQueues to see if that helps, and I'll set up an
>> IO
>> > exception handler.
>> >
>> > Unfortunately the issue can't reliably be reproduced.  We have a cluster
>> of
>> > servers all running their own local embedded queue, and it happens on one
>>
>> > here, one there.
>> >
>> > That said, somebody else already opened a JIRA ticket for what looks like
>> a
>> > similar/nearly identical issue:
>> > https://issues.apache.org/jira/browse/AMQ-3051
>> >
>> > Dan
>> >
>> >
>> > On Thu, Jan 6, 2011 at 9:29 AM, Gary Tully [via ActiveMQ] <
>> > [hidden email] <http://user/SendEmail.jtp?type=node&node=3177622&i=1><[hidden
>> email] <http://user/SendEmail.jtp?type=node&node=3177622&i=2>>
>> >> wrote:
>> >
>> >> One thought, there may be an issue with the default setting of true
>> >> for concurrentStoreAndDispatchQueues that is causing the index to be
>> >> out of sync with the store.
>> >>
>> >> It is worth trying to disable that, use:
>> >> <persistenceAdapter>
>> >>     <kahaDB directory="/usr/local/embedded/activemq-data/kahadb"
>> >> concurrentStoreAndDispatchQueues="false"/>
>> >> </persistenceAdapter>
>> >>
>> >> How often is this occurring, to really get to the bottom of this a
>> >> test case will be needed.
>> >> Does that data file eventually get written, or are there any disk space
>> >> issues?
>> >> It may help to configure an IO exception handler just in case.
>> >> http://activemq.apache.org/configurable-ioexception-handling.html
>> >>
>> >> Would be a good idea to create a jira issue to track this and attach
>> >> what ever log and application information you have to it.
>> >>
>> >> On 6 January 2011 12:39, dcheckoway <[hidden email]<
>> http://user/SendEmail.jtp?type=node&node=3177500&i=0>>
>> >> wrote:
>> >>
>> >> >
>> >> > I'm using an embedded broker (version 5.4.2) with persistence enabled.
>>
>> >>  We're
>> >> > pumping hundreds of millions of messages per day through this thing.
>> >>  Every
>> >> > once in a while, all of a sudden the KahaDB directory starts growing
>> >> > uncontrollably, and these errors start spewing out in the log over
and
>>
>> >> > over...
>> >> >
>> >> > --------------------------
>> >> > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task;
>> >> > AbstractStoreCursor - Failed to fill batch
>> >> > java.lang.RuntimeException: java.io.IOException: Could not locate data
>>
>> >> file
>> >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
>>
>> >>
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
>>
>> >>
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
>>
>> >>
>> >> >        at
>> >> org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
>> >> >        at
>> >> >
>> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
>> >> >        at
>> >> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>>
>> >>
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
>> >> >        at
>> >> >
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>
>> >>
>> >> >        at
>> >> >
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>
>> >>
>> >> >        at java.lang.Thread.run(Thread.java:619)
>> >> > Caused by: java.io.IOException: Could not locate data file
>> >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>> >> >        at
>> org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
>> >>
>> >> >        at org.apache.kahadb.journal.Journal.read(Journal.java:592)
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
>>
>> >>
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
>>
>> >>
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
>>
>> >>
>> >> >        at
>> >> org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
>>
>> >>
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
>>
>> >>
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
>>
>> >>
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
>>
>> >>
>> >> >        ... 10 more
>> >> > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task;
>> Queue
>> >> -
>> >> > Failed to page in more queue messages
>> >> > java.lang.RuntimeException: java.lang.RuntimeException:
>> >> java.io.IOException:
>> >> > Could not locate data file
>> >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:151)
>>
>> >>
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
>>
>> >>
>> >> >        at
>> >> org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
>> >> >        at
>> >> >
>> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
>> >> >        at
>> >> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>>
>> >>
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
>> >> >        at
>> >> >
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>
>> >>
>> >> >        at
>> >> >
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>
>> >>
>> >> >        at java.lang.Thread.run(Thread.java:619)
>> >> > Caused by: java.lang.RuntimeException: java.io.IOException: Could not
>> >> locate
>> >> > data file
>> /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>> >>
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
>>
>> >>
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
>>
>> >>
>> >> >        ... 9 more
>> >> > Caused by: java.io.IOException: Could not locate data file
>> >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>> >> >        at
>> org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
>> >>
>> >> >        at org.apache.kahadb.journal.Journal.read(Journal.java:592)
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
>>
>> >>
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
>>
>> >>
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
>>
>> >>
>> >> >        at
>> >> org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
>>
>> >>
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
>>
>> >>
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
>>
>> >>
>> >> >        at
>> >> >
>> >>
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
>>
>> >>
>> >> >        ... 10 more
>> >> > --------------------------
>> >> >
>> >> > It doesn't happen consistently, but it has happened numerous times.
>>  The
>> >> > only way to "fix" it is to kill the app, blow away the KahaDB data
>> >> > directory, and restart (ugh).
>> >> >
>> >> > Has anybody else seen this issue?  Any idea why it happens?
>> >> >
>> >> > Here's the embedded broker config, fwiw:
>> >> > --------------------------
>> >> >  <broker xmlns="http://activemq.apache.org/schema/core"
>> >> >          id="embeddedActivemqBroker"
>> >> >          useJmx="false"
>> >> >          persistent="true"
>> >> >          dataDirectory="/usr/local/embedded/activemq-data"
>> >> >          schedulerSupport="false"
>> >> >          advisorySupport="false"
>> >> >          enableStatistics="true">
>> >> >    <destinationPolicy>
>> >> >      <policyMap>
>> >> >        <policyEntries>
>> >> >          <policyEntry queue=">" producerFlowControl="false"
>> >> > memoryLimit="20mb"/>
>> >> >        </policyEntries>
>> >> >      </policyMap>
>> >> >    </destinationPolicy>
>> >> >  </broker>
>> >> > --------------------------
>> >> >
>> >> > Help?!
>> >> >
>> >> > -----
>> >> > Dan Checkoway
>> >> > dcheckoway gmail com
>> >> > --
>> >> > View this message in context:
>> >>
>> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t>
>> <
>> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t&by-user=t>>
>>
>> >> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> http://blog.garytully.com
>> >> http://fusesource.com
>> >>
>> >>
>> >> ------------------------------
>> >>  View message @
>> >>
>> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177500.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177500.html?by-user=t>
>> >>
>> >>
>> >>
>> >
>> >
>> > -----
>> > Dan Checkoway
>> > dcheckoway gmail com
>> > --
>> > View this message in context:
>> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177577.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177577.html?by-user=t>
>>
>> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >
>>
>>
>>
>> --
>> http://blog.garytully.com
>> http://fusesource.com
>>
>>
>> ------------------------------
>>  View message @
>> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177622.html
>>
>> To unsubscribe from KahaDB error: "Could not locate data file", click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3177338&code=ZGNoZWNrb3dheUBnbWFpbC5jb218MzE3NzMzOHw3MDc4NzEwMTU=>.
>>
>>
>
>
> -----
> Dan Checkoway
> dcheckoway gmail com
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177745.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Mime
View raw message