Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (athena.apache.org: domain of joe.carter@gmail.com
 designates 209.85.210.171 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAH+vQmPd2biDg1U8bR9mYcPEmR2SkCtsMqyS8x2RwESffe_e9Q@mail.gmail.com>
References: <AANLkTim_NPyHV-gCzcB8pH2m0eyso2vm25p-_byNWHFP@mail.gmail.com>
	<AANLkTi=AjMsTOKT-RidXhtiHoKfGzz7NAgVVo0NveQF+@mail.gmail.com>
	<AANLkTin=bEzOEeCezsXBHpvxgHRArqxdLdsCcTrKBP8N@mail.gmail.com>
	<1311679528547-3695392.post@n4.nabble.com>
	<CAH+vQmPd2biDg1U8bR9mYcPEmR2SkCtsMqyS8x2RwESffe_e9Q@mail.gmail.com>
Date: Wed, 27 Jul 2011 09:39:38 +0100
Message-ID: 
 <CACcs02t9UezKyjkY-67kEb-=fgcp4=4YWFF9jnTy7ehqcUzy2g@mail.gmail.com>
Subject: Re: KahaDB corruption
From: Joe Carter <joe.carter@gmail.com>
To: users@activemq.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks Gary - I'd moved from embedded to a separate broker and had
missed this completely.
For reference I added the following to conf/activemq.xml
                    ignoreMissingJournalfiles=3D"true"
                    checkForCorruptJournalFiles=3D"true"
                    checksumJournalFiles=3D"true" />
to give
        <persistenceAdapter>
            <kahaDB directory=3D"${activemq.base}/data/kahadb"
                    ignoreMissingJournalfiles=3D"true"
                    checkForCorruptJournalFiles=3D"true"
                    checksumJournalFiles=3D"true" />
        </persistenceAdapter>

I pushed the corrupted database back in place and it recovered correctly.
2011-07-27 09:31:19,644 | INFO  | Recovery replayed 2 operations from
the journal in 0.036 seconds. |
org.apache.activemq.store.kahadb.MessageDatabase | main
2011-07-27 09:31:19,649 | INFO  | Detected missing/corrupt journal
files.  Dropped 5 messages from the index in 0.0030 seconds. |
org.apache.activemq.store.kahadb.MessageDatabase | main

So I can confirm the recovery code is working for me.

Thanks.
Joe

On 26 July 2011 12:59, Gary Tully <gary.tully@gmail.com> wrote:
> The flags: checksumJournalFiles, checkForCorruptJournalFiles and
> ignoreMissingJournalfiles are designed to for this use case. Have you
> those enabled?
>
> http://activemq.apache.org/kahadb.html
>
> On 26 July 2011 12:25, JoeC <joe.carter@gmail.com> wrote:
>> I'm currently on 5.5.0 and ran into a different and unrecoverable kahadb
>> case.
>> I ran the system out of diskspace and not unreasonably activemq didn't l=
ike
>> it.
>> After freeing up some space I ran into database corruption as follows.
>> 2011-07-26 10:00:23,316 | INFO =A0| Corrupt journal records found in
>> '/opt/ivb/apache-activemq-5.5.0/data/kahadb/db-326.log' between offsets:
>> 19460423-21031378 | org.apache.kahadb.journal.Journal | main
>> ...
>> 2011-07-26 10:00:23,826 | INFO =A0| Recovering from the journal ... |
>> org.apache.activemq.store.kahadb.MessageDatabase | main
>> 2011-07-26 10:00:23,953 | ERROR | Failed to start ActiveMQ JMS Message
>> Broker. Reason: org.apache.activemq.protobuf.InvalidProtocolBufferExcept=
ion:
>> Protocol message contained an invalid tag (zero). |
>> org.apache.activemq.broker.BrokerService | main
>>
>> Removing the db.data made no difference.
>> I then removed the db-326.log file and restarted twice.
>> The first it complains about not finding db-326.log.
>> The second time is uses a newly created db-1.log.
>>
>> Fortunately this was not a production environment, so the data doesn't
>> matter however I would like a way of recovering the data. This could eve=
n be
>> an offline process.
>> i.e. I quickly reset the database to restore service and then push in th=
e
>> older messages later.
>> My application domain is somewhat tolerant of that approach but it is no=
t
>> tolerant of extended outages.
>> For me, I'd rather (temporarily) lose some data than have a long outage =
so a
>> fully automated recovery is what I'd ideally like irrespective of
>> corruption.
>>
>> Cheers
>> Joe
>>
>>
>> JoeC wrote:
>>>
>>> I've upgraded to 5.4.2 and will let you know how it goes.
>>> I didn't rebuild the index as I've already restarted the process.
>>> In normal operation the queues should be empty for our application so
>>> that was not an issue for me.
>>>
>>> Thanks
>>> Joe
>>>
>>> On 23 February 2011 18:06, Gary Tully &lt;gary.tully@gmail.com&gt; wrot=
e:
>>>> 5.4.2 is better w.r.t abortive shutdown, but for this case, rebuilding
>>>> the index should work.
>>>> remove kahadb/db.data and restart, it will parse the journal to
>>>> rebuild the index.
>>>>
>>>
>>
>>
>> --
>> View this message in context: http://activemq.2283324.n4.nabble.com/Kaha=
DB-corruption-tp3321382p3695392.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
> --
> http://fusesource.com
> http://blog.garytully.com
>