From users-return-28293-apmail-activemq-users-archive=activemq.apache.org@activemq.apache.org Tue Jul 26 11:59:45 2011 Return-Path: X-Original-To: apmail-activemq-users-archive@www.apache.org Delivered-To: apmail-activemq-users-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6E00F792D for ; Tue, 26 Jul 2011 11:59:45 +0000 (UTC) Received: (qmail 94145 invoked by uid 500); 26 Jul 2011 11:59:44 -0000 Delivered-To: apmail-activemq-users-archive@activemq.apache.org Received: (qmail 93811 invoked by uid 500); 26 Jul 2011 11:59:41 -0000 Mailing-List: contact users-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@activemq.apache.org Delivered-To: mailing list users@activemq.apache.org Received: (qmail 93802 invoked by uid 99); 26 Jul 2011 11:59:39 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jul 2011 11:59:39 +0000 X-ASF-Spam-Status: No, hits=0.6 required=5.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,URI_HEX X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of gary.tully@gmail.com designates 209.85.216.171 as permitted sender) Received: from [209.85.216.171] (HELO mail-qy0-f171.google.com) (209.85.216.171) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jul 2011 11:59:34 +0000 Received: by qyl38 with SMTP id 38so1551700qyl.2 for ; Tue, 26 Jul 2011 04:59:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=fC25q9W/EbnHHyVsKzR5/T3ce7GVnhAW6sK+S5K1h0Y=; b=ttzC01cHfi99hHajrAoWbvMpRGAs/aJi8ONJUzPqJV0B/vBVdT+auR+RJHLDGKOfzF PXLVMgMn/ei+fcOp4R4AqijFnR1k1uwvt6KG+UWseVOE9/J9+XqasUD+jzakYxTxbEYz HBBgKFKVAD2p8yoEiBI9UtIxO4bJ9qrgVLI/g= MIME-Version: 1.0 Received: by 10.229.137.19 with SMTP id u19mr4065345qct.173.1311681553935; Tue, 26 Jul 2011 04:59:13 -0700 (PDT) Received: by 10.229.240.130 with HTTP; Tue, 26 Jul 2011 04:59:13 -0700 (PDT) In-Reply-To: <1311679528547-3695392.post@n4.nabble.com> References: <1311679528547-3695392.post@n4.nabble.com> Date: Tue, 26 Jul 2011 12:59:13 +0100 Message-ID: Subject: Re: KahaDB corruption From: Gary Tully To: users@activemq.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable The flags: checksumJournalFiles, checkForCorruptJournalFiles and ignoreMissingJournalfiles are designed to for this use case. Have you those enabled? http://activemq.apache.org/kahadb.html On 26 July 2011 12:25, JoeC wrote: > I'm currently on 5.5.0 and ran into a different and unrecoverable kahadb > case. > I ran the system out of diskspace and not unreasonably activemq didn't li= ke > it. > After freeing up some space I ran into database corruption as follows. > 2011-07-26 10:00:23,316 | INFO =A0| Corrupt journal records found in > '/opt/ivb/apache-activemq-5.5.0/data/kahadb/db-326.log' between offsets: > 19460423-21031378 | org.apache.kahadb.journal.Journal | main > ... > 2011-07-26 10:00:23,826 | INFO =A0| Recovering from the journal ... | > org.apache.activemq.store.kahadb.MessageDatabase | main > 2011-07-26 10:00:23,953 | ERROR | Failed to start ActiveMQ JMS Message > Broker. Reason: org.apache.activemq.protobuf.InvalidProtocolBufferExcepti= on: > Protocol message contained an invalid tag (zero). | > org.apache.activemq.broker.BrokerService | main > > Removing the db.data made no difference. > I then removed the db-326.log file and restarted twice. > The first it complains about not finding db-326.log. > The second time is uses a newly created db-1.log. > > Fortunately this was not a production environment, so the data doesn't > matter however I would like a way of recovering the data. This could even= be > an offline process. > i.e. I quickly reset the database to restore service and then push in the > older messages later. > My application domain is somewhat tolerant of that approach but it is not > tolerant of extended outages. > For me, I'd rather (temporarily) lose some data than have a long outage s= o a > fully automated recovery is what I'd ideally like irrespective of > corruption. > > Cheers > Joe > > > JoeC wrote: >> >> I've upgraded to 5.4.2 and will let you know how it goes. >> I didn't rebuild the index as I've already restarted the process. >> In normal operation the queues should be empty for our application so >> that was not an issue for me. >> >> Thanks >> Joe >> >> On 23 February 2011 18:06, Gary Tully <gary.tully@gmail.com> wrote= : >>> 5.4.2 is better w.r.t abortive shutdown, but for this case, rebuilding >>> the index should work. >>> remove kahadb/db.data and restart, it will parse the journal to >>> rebuild the index. >>> >> > > > -- > View this message in context: http://activemq.2283324.n4.nabble.com/KahaD= B-corruption-tp3321382p3695392.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > --=20 http://fusesource.com http://blog.garytully.com