activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <gary.tu...@gmail.com>
Subject Re: Issues with KahaDB persistence in ActiveMQ 5.6.0 (IndexOutOfBoundsException, IllegalStateException)
Date Mon, 22 Oct 2012 10:37:41 GMT
from the stack trace, it appears you are using the old kaha persistence adapter.
you should be using <kahadb ..>

On 22 October 2012 08:33, Pauli Kaila <pauli.kaila@napa.fi> wrote:
> For the latest release of our product we switched from Apache DerbyDB to
> KahaDB for ActiveMQ persistence. We also upgraded ActiveMQ to 5.6.0. Now two
> of the three production installations, that have been updated to our latest
> version, are experiencing issues with ActiveMQ persistence causing ActiveMQ
> messaging to be flaky at best.
>
> The issues are currently causing our server to be sometimes unable to send
> messages to clients, since we are getting IndexOutOfBoundsExceptions and
> IllegalStateExceptions trying to access inactive pages in KahaDB on our logs
> (detailed stacktraces below).
>
> The product is a client-server product with fat clients. The server and
> clients each have their own internal (as in running in the same JVM) brokers
> with persistence running as a network of brokers. The reason the setup is
> this is that these systems are running on ships, which might have really bad
> internal networks and we are using ActiveMQ to take care that no data is
> lost even if a client is first disconnected from a server and then shutdown.
> The reason why we switched from DerbyDB was that we were having performance
> issues and thought the KahaDB being purpose built would function better.
>
> Since the installations are on ships, we have no direct access to them, so
> we cannot debug them. We are trying to get our hands on the KahaDB files in
> order to see, if we can reproduce the issues in our development environment.
>
> In the mean time I would like to ask, if anyone has bumped in to similar
> issues with ActiveMQ (5.6.0) and KahaDB? And if yes, were you able to solve
> them in any other way than clearing the perisistence store and hoping for
> the best?
>
> -Pauli
>
> An example of an IndexOutOfBoundsException (there are quite many of these in
> the logs):
>
> Caused by: javax.jms.JMSException: Index: 3, Size: 2
>         at
> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1362)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1290)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1785)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:277)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:212)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:74)
> ~[activemq-pool-5.6.0.jar:5.6.0]
>         at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:59)
> ~[activemq-pool-5.6.0.jar:5.6.0]
>         at <product>.messaging.Messenger.send(Messenger.java:168)
> ~[<product>-impl-2012.3.1.jar:2012.3.1]
>         ... 18 common frames omitted
> Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 2
>         at java.util.ArrayList.rangeCheck(Unknown Source) ~[na:1.7.0_07]
>         at java.util.ArrayList.get(Unknown Source) ~[na:1.7.0_07]
>         at
> org.apache.activemq.kaha.impl.index.hash.HashPage.getHashEntry(HashPage.java:181)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.kaha.impl.index.hash.HashPageInfo.getHashEntry(HashPageInfo.java:78)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.kaha.impl.index.hash.HashBin.getHashEntry(HashBin.java:247)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.kaha.impl.index.hash.HashBin.remove(HashBin.java:164)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.kaha.impl.index.hash.HashIndex.remove(HashIndex.java:303)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.kaha.impl.container.MapContainerImpl.remove(MapContainerImpl.java:337)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.kaha.impl.container.MapContainerImpl.place(MapContainerImpl.java:414)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.store.kahadaptor.TopicSubContainer.add(TopicSubContainer.java:62)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.store.kahadaptor.KahaTopicMessageStore.addMessage(KahaTopicMessageStore.java:77)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.store.AbstractMessageStore.addMessage(AbstractMessageStore.java:88)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.store.AbstractMessageStore.asyncAddTopicMessage(AbstractMessageStore.java:105)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.store.ProxyTopicMessageStore.asyncAddTopicMessage(ProxyTopicMessageStore.java:180)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:453)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at org.apache.activemq.broker.region.Topic.send(Topic.java:417)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:407)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:503)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:305)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:306)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:453)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:681)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:231)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         ... 3 common frames omitted
>
> An example of the IllegalStateException (many of these as well):
>
> Caused by: javax.jms.JMSException: Trying to access an inactive page: 969728
>         at
> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1362)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1290)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1785)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:277)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:212)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:74)
> ~[activemq-pool-5.6.0.jar:5.6.0]
>         at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:59)
> ~[activemq-pool-5.6.0.jar:5.6.0]
>         at <product>.messaging.Messenger.send(Messenger.java:168)
> ~[<product>-impl-2012.3.1.jar:2012.3.1]
>         ... 10 common frames omitted
> Caused by: java.lang.IllegalStateException: Trying to access an inactive
> page: 969728
>         at
> org.apache.activemq.kaha.impl.index.hash.HashIndex.lookupPage(HashIndex.java:347)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.kaha.impl.index.hash.HashPageInfo.begin(HashPageInfo.java:96)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.kaha.impl.index.hash.HashBin.getRetrievePage(HashBin.java:269)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.kaha.impl.index.hash.HashBin.getHashEntry(HashBin.java:244)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.kaha.impl.index.hash.HashBin.remove(HashBin.java:164)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.kaha.impl.index.hash.HashIndex.remove(HashIndex.java:303)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.kaha.impl.container.MapContainerImpl.remove(MapContainerImpl.java:337)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.kaha.impl.container.MapContainerImpl.place(MapContainerImpl.java:414)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.store.kahadaptor.TopicSubContainer.add(TopicSubContainer.java:62)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.store.kahadaptor.KahaTopicMessageStore.addMessage(KahaTopicMessageStore.java:77)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.store.AbstractMessageStore.addMessage(AbstractMessageStore.java:88)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.store.AbstractMessageStore.asyncAddTopicMessage(AbstractMessageStore.java:105)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.store.ProxyTopicMessageStore.asyncAddTopicMessage(ProxyTopicMessageStore.java:180)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:453)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at org.apache.activemq.broker.region.Topic.send(Topic.java:417)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:407)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:503)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:305)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:306)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:453)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:681)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:231)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
> ~[activemq-core-5.6.0.jar:5.6.0]
>         ... 3 common frames omitted
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Issues-with-KahaDB-persistence-in-ActiveMQ-5-6-0-IndexOutOfBoundsException-IllegalStateException-tp4658031.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.



-- 
http://redhat.com
http://blog.garytully.com

Mime
View raw message