activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Shannon <christopher.l.shan...@gmail.com>
Subject Re: ActiveMQ failures with strange exception "KahaDB failed to store to Journal" IOException: The parameter is incorrect
Date Wed, 22 Feb 2017 13:42:07 GMT
I'm not sure what caused the exception when reading the index but the
behavior of the broker on IOException is defined by the IOExceptionHandler
that is configured on the broker.  Take a look at DefaultIOExceptionHandler
and all of the available options.  By default on IOException it will shut
down the broker and try and restart it.  It can be configured to ignore
IOExceptions or to disable the restart and just shutdown, or even to kill
the entire JVM, etc.

http://activemq.apache.org/configurable-ioexception-handling.html

As far as why the restart did not work I'm guessing the lock wasn't
released properly on shutdown for some reason.  5.10.0 is pretty old so I'm
not sure there have been any fixes for this already.  I personally set up
my brokers to exit the JVM on IOException and then I have some other
process restart the broker (puppet, etc) so that there is always a clean
JVM on broker start.


On Wed, Feb 22, 2017 at 7:45 AM, Christian Schneider <
chris@die-schneider.net> wrote:

> Has anyone seen this before?
>
> After this exception the broker shuts down and is restarted by the wrapper
> but then we get "kahadb\lock could not be locked as lock is already held
> for this jvm.".
> So there are actually two problems. Why does the broker shut down at all
> and why is the restart not working?
>
> I would be happy about any feedback.
>
> Christian
>
> ------
> 2017-01-30 03:56:26,910 | ERROR | KahaDB failed to store to Journal |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Transport:
> tcp:///10.218.8.223:49837@61616
> java.io.IOException: The parameter is incorrect
>         at java.io.RandomAccessFile.readBytes(Native Method)[:1.7.0_13]
>         at java.io.RandomAccessFile.read(Unknown Source)[:1.7.0_13]
>         at java.io.RandomAccessFile.readFully(Unknown Source)[:1.7.0_13]
>         at java.io.RandomAccessFile.readFully(Unknown Source)[:1.7.0_13]
>         at org.apache.activemq.util.RecoverableRandomAccessFile.readFul
> ly(RecoverableRandomAccessFile.java:75)[activemq-kahadb-stor
> e-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.disk.page.PageFile.readPage
> (PageFile.java:878)[activemq-kahadb-store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.disk.page.Transaction$2.
> readPage(Transaction.java:456)[activemq-kahadb-store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.disk.page.Transaction$2.<
> init>(Transaction.java:447)[activemq-kahadb-store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.disk.page.Transaction.openI
> nputStream(Transaction.java:444)[activemq-kahadb-store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.disk.page.Transaction.load(
> Transaction.java:420)[activemq-kahadb-store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.disk.page.Transaction.load(
> Transaction.java:377)[activemq-kahadb-store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.disk.index.BTreeIndex.loadN
> ode(BTreeIndex.java:262)[activemq-kahadb-store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.disk.index.BTreeNode.getChi
> ld(BTreeNode.java:225)[activemq-kahadb-store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.disk.index.BTreeNode.getLea
> fNode(BTreeNode.java:676)[activemq-kahadb-store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.disk.index.BTreeNode.put(
> BTreeNode.java:369)[activemq-kahadb-store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.disk.index.BTreeIndex.put(
> BTreeIndex.java:189)[activemq-kahadb-store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.MessageDatabase.updateIndex
> (MessageDatabase.java:1304)[activemq-kahadb-store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.MessageDatabase$11.execute(
> MessageDatabase.java:1140)[activemq-kahadb-store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.disk.page.Transaction.execu
> te(Transaction.java:779)[activemq-kahadb-store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.MessageDatabase.process(
> MessageDatabase.java:1137)[activemq-kahadb-store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.MessageDatabase$10.visit(
> MessageDatabase.java:1074)[activemq-kahadb-store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.data.KahaAddMessageCommand.
> visit(KahaAddMessageCommand.java:241)[activemq-kahadb-
> store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.MessageDatabase.process(
> MessageDatabase.java:1071)[activemq-kahadb-store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.MessageDatabase.store(Messa
> geDatabase.java:978)[activemq-kahadb-store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.MessageDatabase.store(Messa
> geDatabase.java:958)[activemq-kahadb-store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageSt
> ore.addMessage(KahaDBStore.java:424)[activemq-kahadb-
> store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.KahaDBTransactionStore.addM
> essage(KahaDBTransactionStore.java:388)[activemq-kahadb-
> store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.store.kahadb.KahaDBTransactionStore$1.
> addMessage(KahaDBTransactionStore.java:160)[activemq-kahadb-
> store-5.10.0.jar:5.10.0]
>         at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.
> java:913)[activemq-broker-5.10.0.jar:5.10.0]
>         at org.apache.activemq.broker.region.Queue.send(Queue.java:733)
> [activemq-broker-5.10.0.jar:5.10.0]
>         at org.apache.activemq.broker.region.AbstractRegion.send(Abstra
> ctRegion.java:424)[activemq-broker-5.10.0.jar:5.10.0]
>         at org.apache.activemq.broker.region.RegionBroker.send(RegionBr
> oker.java:445)[activemq-broker-5.10.0.jar:5.10.0]
>         at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(Mana
> gedRegionBroker.java:297)[activemq-broker-5.10.0.jar:5.10.0]
>         at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.
> java:147)[activemq-broker-5.10.0.jar:5.10.0]
>         at org.apache.activemq.broker.CompositeDestinationBroker.send(C
> ompositeDestinationBroker.java:96)[activemq-broker-5.10.0.jar:5.10.0]
>         at org.apache.activemq.broker.TransactionBroker.send(Transactio
> nBroker.java:307)[activemq-broker-5.10.0.jar:5.10.0]
>         at org.apache.activemq.broker.MutableBrokerFilter.send(MutableB
> rokerFilter.java:152)[activemq-broker-5.10.0.jar:5.10.0]
>         at org.apache.activemq.broker.TransportConnection.processMessag
> e(TransportConnection.java:496)[activemq-broker-5.10.0.jar:5.10.0]
>         at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMe
> ssage.java:756)[activemq-client-5.10.0.jar:5.10.0]
>         at org.apache.activemq.broker.TransportConnection.service(Trans
> portConnection.java:294)[activemq-broker-5.10.0.jar:5.10.0]
>         at org.apache.activemq.broker.TransportConnection$1.onCommand(
> TransportConnection.java:148)[activemq-broker-5.10.0.jar:5.10.0]
>         at org.apache.activemq.transport.MutexTransport.onCommand(Mutex
> Transport.java:50)[activemq-client-5.10.0.jar:5.10.0]
>         at org.apache.activemq.transport.WireFormatNegotiator.onCommand
> (WireFormatNegotiator.java:113)[activemq-client-5.10.0.jar:5.10.0]
>         at org.apache.activemq.transport.AbstractInactivityMonitor.onCo
> mmand(AbstractInactivityMonitor.java:270)[activemq-client-5.
> 10.0.jar:5.10.0]
>         at org.apache.activemq.transport.TransportSupport.doConsume(Tra
> nsportSupport.java:83)[activemq-client-5.10.0.jar:5.10.0]
>         at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTran
> sport.java:214)[activemq-client-5.10.0.jar:5.10.0]
>         at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransp
> ort.java:196)[activemq-client-5.10.0.jar:5.10.0]
>         at java.lang.Thread.run(Unknown Source)[:1.7.0_13]
>
> --
> Christian Schneider
> http://www.liquid-reality.de
>
> Open Source Architect
> http://www.talend.com
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message