Return-Path: X-Original-To: apmail-activemq-dev-archive@www.apache.org Delivered-To: apmail-activemq-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 17622E638 for ; Wed, 6 Mar 2013 14:52:16 +0000 (UTC) Received: (qmail 43978 invoked by uid 500); 6 Mar 2013 14:52:14 -0000 Delivered-To: apmail-activemq-dev-archive@activemq.apache.org Received: (qmail 43793 invoked by uid 500); 6 Mar 2013 14:52:14 -0000 Mailing-List: contact dev-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@activemq.apache.org Delivered-To: mailing list dev@activemq.apache.org Received: (qmail 43450 invoked by uid 99); 6 Mar 2013 14:52:13 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Mar 2013 14:52:13 +0000 Date: Wed, 6 Mar 2013 14:52:13 +0000 (UTC) From: "Timothy Bish (JIRA)" To: dev@activemq.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (AMQ-4339) Corrupt KahaDB Journal may cause EOFException at Broker startup MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/AMQ-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594750#comment-13594750 ] Timothy Bish commented on AMQ-4339: ----------------------------------- I had to use the Eclipse runner because surefire kept swallowing the output for some reason. > Corrupt KahaDB Journal may cause EOFException at Broker startup > --------------------------------------------------------------- > > Key: AMQ-4339 > URL: https://issues.apache.org/jira/browse/AMQ-4339 > Project: ActiveMQ > Issue Type: Bug > Components: Message Store > Affects Versions: 5.8.0 > Environment: Java 1.6.0 most releases, tested on lots of different hardware, tested on Linux distros only. > Reporter: Ken Barber > Labels: kahaDB > Attachments: AMQ-4339-fix2.patch, AMQ-4339-fix.patch, AMQ-4339-test-case.patch > > > During the occasional KahaDB journal corruption, ActiveMQ may receive an EOFException. The easiest way to replicate this for us has been to purposely put junk into a journal as so: > echo "asdf" > db-1.log > The exception in this case has been (this error is specific to AMQ 5.6.0, so lines may mismatch on 5.8.0, but similar problem was confirmed on 5.8.0): > {code} > 2013-02-13 11:35:27,465 ERROR [main] [broker.BrokerService] Failed to start Apache ActiveMQ (localhost, null). Reason: java.io.EOFException > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:776) > at org.apache.activemq.store.kahadb.disk.journal.DataFileAccessor.readRecord(DataFileAccessor.java:81) > at org.apache.activemq.store.kahadb.disk.journal.Journal.read(Journal.java:604) > at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:961) > at org.apache.activemq.store.kahadb.MessageDatabase.recoverProducerAudit(MessageDatabase.java:629) > at org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:555) > at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:369) > at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:387) > at org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:240) > at org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:180) > at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55) > at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStart(KahaDBPersistenceAdapter.java:220) > at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55) > at org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:598) > at org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:587) > at org.apache.activemq.broker.BrokerService.start(BrokerService.java:552) > at com.puppetlabs.mq$start_broker_BANG_.invoke(mq.clj:112) > at com.puppetlabs.puppetdb.cli.services$_main.doInvoke(services.clj:374) > at clojure.lang.RestFn.invoke(RestFn.java:421) > at clojure.lang.Var.invoke(Var.java:419) > at clojure.lang.AFn.applyToHelper(AFn.java:163) > at clojure.lang.Var.applyTo(Var.java:532) > at clojure.core$apply.invoke(core.clj:601) > at com.puppetlabs.puppetdb.core$_main.doInvoke(core.clj:79) > at clojure.lang.RestFn.applyTo(RestFn.java:137) > at com.puppetlabs.puppetdb.core.main(Unknown Source) > 2013-02-13 11:35:27,983 ERROR [main] [puppetlabs.utils] Uncaught exception > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:776) > at org.apache.activemq.store.kahadb.disk.journal.DataFileAccessor.readRecord(DataFileAccessor.java:81) > at org.apache.activemq.store.kahadb.disk.journal.Journal.read(Journal.java:604) > at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:961) > at org.apache.activemq.store.kahadb.MessageDatabase.recoverProducerAudit(MessageDatabase.java:629) > at org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:555) > at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:369) > at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:387) > at org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:240) > at org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:180) > at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55) > at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStart(KahaDBPersistenceAdapter.java:220) > at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55) > at org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:598) > at org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:587) > at org.apache.activemq.broker.BrokerService.start(BrokerService.java:552) > at com.puppetlabs.mq$start_broker_BANG_.invoke(mq.clj:112) > at com.puppetlabs.puppetdb.cli.services$_main.doInvoke(services.clj:374) > at clojure.lang.RestFn.invoke(RestFn.java:421) > at clojure.lang.Var.invoke(Var.java:419) > at clojure.lang.AFn.applyToHelper(AFn.java:163) > at clojure.lang.Var.applyTo(Var.java:532) > at clojure.core$apply.invoke(core.clj:601) > at com.puppetlabs.puppetdb.core$_main.doInvoke(core.clj:79) > at clojure.lang.RestFn.applyTo(RestFn.java:137) > at com.puppetlabs.puppetdb.core.main(Unknown Source) > {code} > What is strange about this problem, is that upon a restart of the broker the journal is 'reset' it would seem and things go fine afterwards. > I just want to stress that the corruption wasn't caused by KahaDB, at least we haven't seen any cases. The corruptions were either caused by: disk filling up, or bad copies when migrating directories etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira