activemq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vidyashree (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQ-6663) IO exceptions causing Broker Service shutdown; When broker restarts, active-mq ignores the configured location for kahadb
Date Tue, 02 May 2017 15:28:04 GMT

    [ https://issues.apache.org/jira/browse/AMQ-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15993097#comment-15993097
] 

Vidyashree commented on AMQ-6663:
---------------------------------

Hi Christopher,
Thanks for the comments. 
As of now this issue has been observed in production couple of times. But the root cause of
index corruption is still unknown to us. 
After the occurrence of this IO Exception, ActiveMQ DefaultIOExceptionHanlder tried to stop
and restart the broker service. During this restart of broker service, it did not pick the
configured data directory path from the activemq.xml configuration that we have defined. 
>From the BrokerService code we can see the below snippet:
 public File getDataDirectoryFile() {
        if (dataDirectoryFile == null) {
            dataDirectoryFile = new File(IOHelper.getDefaultDataDirectory());
        }
        return dataDirectoryFile;
    }
In our case, this dataDirectoryFile value becomes null during restarting the broker service
during runtime(even if the configured path exists). Please let us know why did this happen.

> IO exceptions causing Broker Service shutdown; When broker restarts, active-mq ignores
the configured location for kahadb
> -------------------------------------------------------------------------------------------------------------------------
>
>                 Key: AMQ-6663
>                 URL: https://issues.apache.org/jira/browse/AMQ-6663
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker, KahaDB, Message Store
>    Affects Versions: 5.11.0
>            Reporter: Vidyashree
>            Priority: Critical
>
> ActiveMQ version: 5.11.0.redhat-620143
> A-MQ configuration in activemq.xml
> 	<bean id="persistenceAdapter"
> 		class="org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter">
> 		<property name="directory" value="${activemq.home}/${id}/jms/kahaDb"/>
> 		<property name="enableJournalDiskSyncs" value="false" />
> 		<property name="journalMaxFileLength" value="33554432" />
> 		<property name="ignoreMissingJournalfiles" value="true"/>
> 		<property name="checkForCorruptJournalFiles" value="true"/>
> 	</bean>
> <bean id="brokerServices"
> 		class="org.apache.activemq.broker.BrokerService">
> 		<description>AMQ Broker</description>
> 		<property name="dataDirectory" value="${activemq.home}/${id}/jms"/>
> 		<property name="systemUsage" ref="systemUsage" />
> 		<property name="persistenceAdapter" ref="persistenceAdapter" />
> 		<property name="managementContext" ref="managementContext" />
> 		<property name="destinationPolicy" ref="destinationPolicy"/>
> 		<property name="useShutdownHook" value="false"/>
> 	</bean>
> Problem description: 
> During durable subscription removal, we are observing IO exceptions as below, and it
is resulting in shutdown of the broker. And we also observe change in path of the kahadb location
to the active-mq default location({$home}/activemq-data/activemq-instance1/KahaDB). It should
have taken the configured location from activemq.xml
> Stack trace from the EOFException :
> 2017-04-11 04:56:01,640 org.apache.activemq.store.kahadb.MessageDatabase ERROR (MessageDatabase.java:store:1010)
[CamelJettyServer(0x57063e08)-3734787] [3734787] | KahaDB failed to store to Journal
> java.io.EOFException
> 	at java.io.DataInputStream.readFully(Unknown Source)
> 	at java.io.DataInputStream.readLong(Unknown Source)
> 	at org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1988)
> 	at org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1910)
> 	at org.apache.activemq.store.kahadb.disk.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:209)
> 	at org.apache.activemq.store.kahadb.disk.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:155)
> 	at org.apache.activemq.store.kahadb.disk.page.Transaction.load(Transaction.java:422)
> 	at org.apache.activemq.store.kahadb.disk.page.Transaction.load(Transaction.java:377)
> 	at org.apache.activemq.store.kahadb.disk.index.BTreeIndex.loadNode(BTreeIndex.java:266)
> 	at org.apache.activemq.store.kahadb.disk.index.BTreeIndex.getRoot(BTreeIndex.java:174)
> 	at org.apache.activemq.store.kahadb.disk.index.BTreeIndex.remove(BTreeIndex.java:194)
> 	at org.apache.activemq.store.kahadb.MessageDatabase.updateIndex(MessageDatabase.java:1469)
> 	at org.apache.activemq.store.kahadb.MessageDatabase.updateIndex(MessageDatabase.java:1500)
> 	at org.apache.activemq.store.kahadb.MessageDatabase$15.execute(MessageDatabase.java:1219)
> 	at org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)
> 	at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:1216)
> 	at org.apache.activemq.store.kahadb.MessageDatabase$10.visit(MessageDatabase.java:1115)
> 	at org.apache.activemq.store.kahadb.data.KahaSubscriptionCommand.visit(KahaSubscriptionCommand.java:187)
> 	at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:1082)
> 	at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:989)
> 	at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:969)
> 	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBTopicMessageStore.deleteSubscription(KahaDBStore.java:807)
> 	at org.apache.activemq.store.ProxyTopicMessageStore.deleteSubscription(ProxyTopicMessageStore.java:103)
> 	at org.apache.activemq.broker.region.Topic.deleteSubscription(Topic.java:201)
> 	at org.apache.activemq.broker.region.TopicRegion.removeSubscription(TopicRegion.java:220)
> 	at org.apache.activemq.broker.region.RegionBroker.removeSubscription(RegionBroker.java:448)
> 	at org.apache.activemq.broker.BrokerFilter.removeSubscription(BrokerFilter.java:117)
> 	at org.apache.activemq.advisory.AdvisoryBroker.removeSubscription(AdvisoryBroker.java:314)
> 	at org.apache.activemq.broker.BrokerFilter.removeSubscription(BrokerFilter.java:117)
> 	at org.apache.activemq.broker.BrokerFilter.removeSubscription(BrokerFilter.java:117)
> 	at org.apache.activemq.broker.MutableBrokerFilter.removeSubscription(MutableBrokerFilter.java:122)
> 	at org.apache.activemq.broker.jmx.BrokerView.destroyDurableSubscriber(BrokerView.java:449)
> 	at org.apache.camel.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:63)
> 	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
> 	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
> 	at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
> 	at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
> 	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
> 	at org.apache.camel.component.jetty.CamelContinuationServlet.service(CamelContinuationServlet.java:150)
> 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> 	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
> 	at org.eclipse.jetty.servlets.MultiPartFilter.doFilter(MultiPartFilter.java:131)
> 	at org.apache.camel.component.jetty.CamelFilterWrapper.doFilter(CamelFilterWrapper.java:43)
> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> 	at org.eclipse.jetty.server.Server.handle(Server.java:370)
> 	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
> 	at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)
> 	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)
> 	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
> 	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
> 	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> 	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
> 	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> 	at java.lang.Thread.run(Unknown Source)
> instance1 2017-04-11 04:56:01,646 org.apache.activemq.util.DefaultIOExceptionHandler
 INFO (DefaultIOExceptionHandler.java:stopBroker:165) [CamelJettyServer(0x57063e08)-3734787]
[3734787] | Stopping BrokerService[activemq-instance1] due to exception, java.io.EOFException
> java.io.EOFException
> 	at java.io.DataInputStream.readFully(Unknown Source)
> 	at java.io.DataInputStream.readLong(Unknown Source)
> 	at org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1988)
> 	
> 1. Why did ActiveMQ consider the default location for KahaDB (Mentioned in IOHelper and
MessageDatabase classes of activemq) instead of the configured path during broker restart?
> 2. When we restarted the system with the affected contents of kahaDB store, we found
the below information from the active mq log
> org.apache.activemq.store.kahadb.MessageDatabase  WARN (MessageDatabase.java:open:382)
[main] [1] | Index corrupted. Recovering the index through journal replay. Cause:java.io.EOFException
> org.apache.activemq.store.kahadb.MessageDatabase DEBUG (MessageDatabase.java:open:384)
[main] [1] | Index load failure
> 	- What could be the cause of index corruption?
> 	- What happens if there is index corruption in the KahaDB store and system does not
restart?
> 	- We did not see any issue with disk such as disk is unavailable of no more space on
the disk. So, in these conditions, how did the corruption happened in the system?
> Is it related to https://issues.apache.org/jira/browse/AMQ-6086?	



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message