activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dcheckoway <dchecko...@gmail.com>
Subject Re: KahaDB error: "Could not locate data file"
Date Thu, 06 Jan 2011 16:47:29 GMT

Yeah, we're using whatever is the default persistence provider in 5.4.2.  I
opened a fresh ticket for it:

https://issues.apache.org/jira/browse/AMQ-3120

Thanks,
Dan

On Thu, Jan 6, 2011 at 10:42 AM, Gary Tully [via ActiveMQ] <
ml-node+3177622-1458695149-140184@n4.nabble.com<ml-node%2B3177622-1458695149-140184@n4.nabble.com>
> wrote:

> AMQ-3051 is using the older AMQPersistenceAdapter and NFS, so it seems
> to be different from your use case?
>
> On 6 January 2011 15:14, dcheckoway <[hidden email]<http://user/SendEmail.jtp?type=node&node=3177622&i=0>>
> wrote:
>
> >
> > Gary,
> >
> > Thanks for the reply and advice.  I'll definitely try disabling
> > concurrentStoreAndDispatchQueues to see if that helps, and I'll set up an
> IO
> > exception handler.
> >
> > Unfortunately the issue can't reliably be reproduced.  We have a cluster
> of
> > servers all running their own local embedded queue, and it happens on one
>
> > here, one there.
> >
> > That said, somebody else already opened a JIRA ticket for what looks like
> a
> > similar/nearly identical issue:
> > https://issues.apache.org/jira/browse/AMQ-3051
> >
> > Dan
> >
> >
> > On Thu, Jan 6, 2011 at 9:29 AM, Gary Tully [via ActiveMQ] <
> > [hidden email] <http://user/SendEmail.jtp?type=node&node=3177622&i=1><[hidden
> email] <http://user/SendEmail.jtp?type=node&node=3177622&i=2>>
> >> wrote:
> >
> >> One thought, there may be an issue with the default setting of true
> >> for concurrentStoreAndDispatchQueues that is causing the index to be
> >> out of sync with the store.
> >>
> >> It is worth trying to disable that, use:
> >> <persistenceAdapter>
> >>     <kahaDB directory="/usr/local/embedded/activemq-data/kahadb"
> >> concurrentStoreAndDispatchQueues="false"/>
> >> </persistenceAdapter>
> >>
> >> How often is this occurring, to really get to the bottom of this a
> >> test case will be needed.
> >> Does that data file eventually get written, or are there any disk space
> >> issues?
> >> It may help to configure an IO exception handler just in case.
> >> http://activemq.apache.org/configurable-ioexception-handling.html
> >>
> >> Would be a good idea to create a jira issue to track this and attach
> >> what ever log and application information you have to it.
> >>
> >> On 6 January 2011 12:39, dcheckoway <[hidden email]<
> http://user/SendEmail.jtp?type=node&node=3177500&i=0>>
> >> wrote:
> >>
> >> >
> >> > I'm using an embedded broker (version 5.4.2) with persistence enabled.
>
> >>  We're
> >> > pumping hundreds of millions of messages per day through this thing.
> >>  Every
> >> > once in a while, all of a sudden the KahaDB directory starts growing
> >> > uncontrollably, and these errors start spewing out in the log over and
>
> >> > over...
> >> >
> >> > --------------------------
> >> > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task;
> >> > AbstractStoreCursor - Failed to fill batch
> >> > java.lang.RuntimeException: java.io.IOException: Could not locate data
>
> >> file
> >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> >> >        at
> >> >
> >>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
>
> >>
> >> >        at
> >> >
> >>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
>
> >>
> >> >        at
> >> >
> >>
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
>
> >>
> >> >        at
> >> org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
> >> >        at
> >> >
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
> >> >        at
> >> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
> >> >        at
> >> >
> >>
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>
> >>
> >> >        at
> >> >
> >>
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
> >> >        at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> >>
> >> >        at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> >>
> >> >        at java.lang.Thread.run(Thread.java:619)
> >> > Caused by: java.io.IOException: Could not locate data file
> >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> >> >        at
> org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
> >>
> >> >        at org.apache.kahadb.journal.Journal.read(Journal.java:592)
> >> >        at
> >> >
> >>
> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
>
> >>
> >> >        at
> >> >
> >>
> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
>
> >>
> >> >        at
> >> >
> >>
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
>
> >>
> >> >        at
> >> org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> >> >        at
> >> >
> >>
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
>
> >>
> >> >        at
> >> >
> >>
> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
>
> >>
> >> >        at
> >> >
> >>
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
>
> >>
> >> >        at
> >> >
> >>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
>
> >>
> >> >        ... 10 more
> >> > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task;
> Queue
> >> -
> >> > Failed to page in more queue messages
> >> > java.lang.RuntimeException: java.lang.RuntimeException:
> >> java.io.IOException:
> >> > Could not locate data file
> >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> >> >        at
> >> >
> >>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:151)
>
> >>
> >> >        at
> >> >
> >>
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
>
> >>
> >> >        at
> >> org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
> >> >        at
> >> >
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
> >> >        at
> >> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
> >> >        at
> >> >
> >>
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>
> >>
> >> >        at
> >> >
> >>
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
> >> >        at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> >>
> >> >        at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> >>
> >> >        at java.lang.Thread.run(Thread.java:619)
> >> > Caused by: java.lang.RuntimeException: java.io.IOException: Could not
> >> locate
> >> > data file
> /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> >>
> >> >        at
> >> >
> >>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
>
> >>
> >> >        at
> >> >
> >>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
>
> >>
> >> >        ... 9 more
> >> > Caused by: java.io.IOException: Could not locate data file
> >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> >> >        at
> org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
> >>
> >> >        at org.apache.kahadb.journal.Journal.read(Journal.java:592)
> >> >        at
> >> >
> >>
> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
>
> >>
> >> >        at
> >> >
> >>
> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
>
> >>
> >> >        at
> >> >
> >>
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
>
> >>
> >> >        at
> >> org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> >> >        at
> >> >
> >>
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
>
> >>
> >> >        at
> >> >
> >>
> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
>
> >>
> >> >        at
> >> >
> >>
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
>
> >>
> >> >        at
> >> >
> >>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
>
> >>
> >> >        ... 10 more
> >> > --------------------------
> >> >
> >> > It doesn't happen consistently, but it has happened numerous times.
>  The
> >> > only way to "fix" it is to kill the app, blow away the KahaDB data
> >> > directory, and restart (ugh).
> >> >
> >> > Has anybody else seen this issue?  Any idea why it happens?
> >> >
> >> > Here's the embedded broker config, fwiw:
> >> > --------------------------
> >> >  <broker xmlns="http://activemq.apache.org/schema/core"
> >> >          id="embeddedActivemqBroker"
> >> >          useJmx="false"
> >> >          persistent="true"
> >> >          dataDirectory="/usr/local/embedded/activemq-data"
> >> >          schedulerSupport="false"
> >> >          advisorySupport="false"
> >> >          enableStatistics="true">
> >> >    <destinationPolicy>
> >> >      <policyMap>
> >> >        <policyEntries>
> >> >          <policyEntry queue=">" producerFlowControl="false"
> >> > memoryLimit="20mb"/>
> >> >        </policyEntries>
> >> >      </policyMap>
> >> >    </destinationPolicy>
> >> >  </broker>
> >> > --------------------------
> >> >
> >> > Help?!
> >> >
> >> > -----
> >> > Dan Checkoway
> >> > dcheckoway gmail com
> >> > --
> >> > View this message in context:
> >>
> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t>
> <
> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t&by-user=t>>
>
> >> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >> >
> >>
> >>
> >>
> >> --
> >> http://blog.garytully.com
> >> http://fusesource.com
> >>
> >>
> >> ------------------------------
> >>  View message @
> >>
> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177500.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177500.html?by-user=t>
> >>
> >>
> >>
> >
> >
> > -----
> > Dan Checkoway
> > dcheckoway gmail com
> > --
> > View this message in context:
> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177577.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177577.html?by-user=t>
>
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
>
>
> --
> http://blog.garytully.com
> http://fusesource.com
>
>
> ------------------------------
>  View message @
> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177622.html
>
> To unsubscribe from KahaDB error: "Could not locate data file", click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3177338&code=ZGNoZWNrb3dheUBnbWFpbC5jb218MzE3NzMzOHw3MDc4NzEwMTU=>.
>
>


-----
Dan Checkoway
dcheckoway gmail com
-- 
View this message in context: http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177745.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message