Return-Path: X-Original-To: apmail-activemq-users-archive@www.apache.org Delivered-To: apmail-activemq-users-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0417CD866 for ; Mon, 22 Oct 2012 10:38:09 +0000 (UTC) Received: (qmail 76246 invoked by uid 500); 22 Oct 2012 10:38:08 -0000 Delivered-To: apmail-activemq-users-archive@activemq.apache.org Received: (qmail 75936 invoked by uid 500); 22 Oct 2012 10:38:08 -0000 Mailing-List: contact users-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@activemq.apache.org Delivered-To: mailing list users@activemq.apache.org Received: (qmail 75881 invoked by uid 99); 22 Oct 2012 10:38:07 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Oct 2012 10:38:07 +0000 X-ASF-Spam-Status: No, hits=0.6 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS,URI_HEX X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of gary.tully@gmail.com designates 209.85.212.179 as permitted sender) Received: from [209.85.212.179] (HELO mail-wi0-f179.google.com) (209.85.212.179) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Oct 2012 10:38:02 +0000 Received: by mail-wi0-f179.google.com with SMTP id hq7so1990420wib.14 for ; Mon, 22 Oct 2012 03:37:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=n3+bzGh6zZvujpm4xjNYoQTt974VLhCazImuaScnCiQ=; b=DsE9thdjC99UjhxXwo5sWHmRJI9t9Cs9xaHlCinNVeOAFTmcNxo1wA9zFjolKEvXQa Yj4jRKRjsCdHSJmj1RoNYF4x2pcIG3aZo8lg2FQ7TMMoTuXsdQ3t7w68Xn204gLM5hju GV1ynezaKpmd90qhVBjWFGryIOsHggkFuoJ3KicKK1gVE98JTHJpYyBMrHTB/07OgPn0 OEof/33ABZ4vZ3e9LY4tJcvgn+KlCs7gUSaB6t2mF1eGZk2NSlc22t4zzJ3npNtR55GT HTfoR4TefgsnD0taPtkLU6msa2gtsEdpA1EJSJovNNI49JOEFfsrVEuy/q5Qinp1QP15 IfoQ== MIME-Version: 1.0 Received: by 10.180.96.164 with SMTP id dt4mr35894090wib.10.1350902261194; Mon, 22 Oct 2012 03:37:41 -0700 (PDT) Received: by 10.194.62.197 with HTTP; Mon, 22 Oct 2012 03:37:41 -0700 (PDT) In-Reply-To: <1350891205152-4658031.post@n4.nabble.com> References: <1350891205152-4658031.post@n4.nabble.com> Date: Mon, 22 Oct 2012 11:37:41 +0100 Message-ID: Subject: Re: Issues with KahaDB persistence in ActiveMQ 5.6.0 (IndexOutOfBoundsException, IllegalStateException) From: Gary Tully To: users@activemq.apache.org Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Checked: Checked by ClamAV on apache.org from the stack trace, it appears you are using the old kaha persistence adapter. you should be using On 22 October 2012 08:33, Pauli Kaila wrote: > For the latest release of our product we switched from Apache DerbyDB to > KahaDB for ActiveMQ persistence. We also upgraded ActiveMQ to 5.6.0. Now two > of the three production installations, that have been updated to our latest > version, are experiencing issues with ActiveMQ persistence causing ActiveMQ > messaging to be flaky at best. > > The issues are currently causing our server to be sometimes unable to send > messages to clients, since we are getting IndexOutOfBoundsExceptions and > IllegalStateExceptions trying to access inactive pages in KahaDB on our logs > (detailed stacktraces below). > > The product is a client-server product with fat clients. The server and > clients each have their own internal (as in running in the same JVM) brokers > with persistence running as a network of brokers. The reason the setup is > this is that these systems are running on ships, which might have really bad > internal networks and we are using ActiveMQ to take care that no data is > lost even if a client is first disconnected from a server and then shutdown. > The reason why we switched from DerbyDB was that we were having performance > issues and thought the KahaDB being purpose built would function better. > > Since the installations are on ships, we have no direct access to them, so > we cannot debug them. We are trying to get our hands on the KahaDB files in > order to see, if we can reproduce the issues in our development environment. > > In the mean time I would like to ask, if anyone has bumped in to similar > issues with ActiveMQ (5.6.0) and KahaDB? And if yes, were you able to solve > them in any other way than clearing the perisistence store and hoping for > the best? > > -Pauli > > An example of an IndexOutOfBoundsException (there are quite many of these in > the logs): > > Caused by: javax.jms.JMSException: Index: 3, Size: 2 > at > org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1362) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1290) > ~[activemq-core-5.6.0.jar:5.6.0] > at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1785) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:277) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:212) > ~[activemq-core-5.6.0.jar:5.6.0] > at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:74) > ~[activemq-pool-5.6.0.jar:5.6.0] > at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:59) > ~[activemq-pool-5.6.0.jar:5.6.0] > at .messaging.Messenger.send(Messenger.java:168) > ~[-impl-2012.3.1.jar:2012.3.1] > ... 18 common frames omitted > Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 2 > at java.util.ArrayList.rangeCheck(Unknown Source) ~[na:1.7.0_07] > at java.util.ArrayList.get(Unknown Source) ~[na:1.7.0_07] > at > org.apache.activemq.kaha.impl.index.hash.HashPage.getHashEntry(HashPage.java:181) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.index.hash.HashPageInfo.getHashEntry(HashPageInfo.java:78) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.index.hash.HashBin.getHashEntry(HashBin.java:247) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.index.hash.HashBin.remove(HashBin.java:164) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.index.hash.HashIndex.remove(HashIndex.java:303) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.container.MapContainerImpl.remove(MapContainerImpl.java:337) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.container.MapContainerImpl.place(MapContainerImpl.java:414) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.store.kahadaptor.TopicSubContainer.add(TopicSubContainer.java:62) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.store.kahadaptor.KahaTopicMessageStore.addMessage(KahaTopicMessageStore.java:77) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.store.AbstractMessageStore.addMessage(AbstractMessageStore.java:88) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.store.AbstractMessageStore.asyncAddTopicMessage(AbstractMessageStore.java:105) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.store.ProxyTopicMessageStore.asyncAddTopicMessage(ProxyTopicMessageStore.java:180) > ~[activemq-core-5.6.0.jar:5.6.0] > at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:453) > ~[activemq-core-5.6.0.jar:5.6.0] > at org.apache.activemq.broker.region.Topic.send(Topic.java:417) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:407) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:503) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:305) > ~[activemq-core-5.6.0.jar:5.6.0] > at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:306) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:453) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:681) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:231) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43) > ~[activemq-core-5.6.0.jar:5.6.0] > ... 3 common frames omitted > > An example of the IllegalStateException (many of these as well): > > Caused by: javax.jms.JMSException: Trying to access an inactive page: 969728 > at > org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1362) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1290) > ~[activemq-core-5.6.0.jar:5.6.0] > at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1785) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:277) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:212) > ~[activemq-core-5.6.0.jar:5.6.0] > at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:74) > ~[activemq-pool-5.6.0.jar:5.6.0] > at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:59) > ~[activemq-pool-5.6.0.jar:5.6.0] > at .messaging.Messenger.send(Messenger.java:168) > ~[-impl-2012.3.1.jar:2012.3.1] > ... 10 common frames omitted > Caused by: java.lang.IllegalStateException: Trying to access an inactive > page: 969728 > at > org.apache.activemq.kaha.impl.index.hash.HashIndex.lookupPage(HashIndex.java:347) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.index.hash.HashPageInfo.begin(HashPageInfo.java:96) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.index.hash.HashBin.getRetrievePage(HashBin.java:269) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.index.hash.HashBin.getHashEntry(HashBin.java:244) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.index.hash.HashBin.remove(HashBin.java:164) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.index.hash.HashIndex.remove(HashIndex.java:303) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.container.MapContainerImpl.remove(MapContainerImpl.java:337) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.kaha.impl.container.MapContainerImpl.place(MapContainerImpl.java:414) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.store.kahadaptor.TopicSubContainer.add(TopicSubContainer.java:62) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.store.kahadaptor.KahaTopicMessageStore.addMessage(KahaTopicMessageStore.java:77) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.store.AbstractMessageStore.addMessage(AbstractMessageStore.java:88) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.store.AbstractMessageStore.asyncAddTopicMessage(AbstractMessageStore.java:105) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.store.ProxyTopicMessageStore.asyncAddTopicMessage(ProxyTopicMessageStore.java:180) > ~[activemq-core-5.6.0.jar:5.6.0] > at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:453) > ~[activemq-core-5.6.0.jar:5.6.0] > at org.apache.activemq.broker.region.Topic.send(Topic.java:417) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:407) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:503) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:305) > ~[activemq-core-5.6.0.jar:5.6.0] > at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:306) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:453) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:681) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:231) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) > ~[activemq-core-5.6.0.jar:5.6.0] > at > org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43) > ~[activemq-core-5.6.0.jar:5.6.0] > ... 3 common frames omitted > > > > > -- > View this message in context: http://activemq.2283324.n4.nabble.com/Issues-with-KahaDB-persistence-in-ActiveMQ-5-6-0-IndexOutOfBoundsException-IllegalStateException-tp4658031.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. -- http://redhat.com http://blog.garytully.com