activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Carter <joe.car...@gmail.com>
Subject Re: KahaDB corruption
Date Wed, 27 Jul 2011 08:39:38 GMT
Thanks Gary - I'd moved from embedded to a separate broker and had
missed this completely.
For reference I added the following to conf/activemq.xml
                    ignoreMissingJournalfiles="true"
                    checkForCorruptJournalFiles="true"
                    checksumJournalFiles="true" />
to give
        <persistenceAdapter>
            <kahaDB directory="${activemq.base}/data/kahadb"
                    ignoreMissingJournalfiles="true"
                    checkForCorruptJournalFiles="true"
                    checksumJournalFiles="true" />
        </persistenceAdapter>

I pushed the corrupted database back in place and it recovered correctly.
2011-07-27 09:31:19,644 | INFO  | Recovery replayed 2 operations from
the journal in 0.036 seconds. |
org.apache.activemq.store.kahadb.MessageDatabase | main
2011-07-27 09:31:19,649 | INFO  | Detected missing/corrupt journal
files.  Dropped 5 messages from the index in 0.0030 seconds. |
org.apache.activemq.store.kahadb.MessageDatabase | main

So I can confirm the recovery code is working for me.

Thanks.
Joe

On 26 July 2011 12:59, Gary Tully <gary.tully@gmail.com> wrote:
> The flags: checksumJournalFiles, checkForCorruptJournalFiles and
> ignoreMissingJournalfiles are designed to for this use case. Have you
> those enabled?
>
> http://activemq.apache.org/kahadb.html
>
> On 26 July 2011 12:25, JoeC <joe.carter@gmail.com> wrote:
>> I'm currently on 5.5.0 and ran into a different and unrecoverable kahadb
>> case.
>> I ran the system out of diskspace and not unreasonably activemq didn't like
>> it.
>> After freeing up some space I ran into database corruption as follows.
>> 2011-07-26 10:00:23,316 | INFO  | Corrupt journal records found in
>> '/opt/ivb/apache-activemq-5.5.0/data/kahadb/db-326.log' between offsets:
>> 19460423-21031378 | org.apache.kahadb.journal.Journal | main
>> ...
>> 2011-07-26 10:00:23,826 | INFO  | Recovering from the journal ... |
>> org.apache.activemq.store.kahadb.MessageDatabase | main
>> 2011-07-26 10:00:23,953 | ERROR | Failed to start ActiveMQ JMS Message
>> Broker. Reason: org.apache.activemq.protobuf.InvalidProtocolBufferException:
>> Protocol message contained an invalid tag (zero). |
>> org.apache.activemq.broker.BrokerService | main
>>
>> Removing the db.data made no difference.
>> I then removed the db-326.log file and restarted twice.
>> The first it complains about not finding db-326.log.
>> The second time is uses a newly created db-1.log.
>>
>> Fortunately this was not a production environment, so the data doesn't
>> matter however I would like a way of recovering the data. This could even be
>> an offline process.
>> i.e. I quickly reset the database to restore service and then push in the
>> older messages later.
>> My application domain is somewhat tolerant of that approach but it is not
>> tolerant of extended outages.
>> For me, I'd rather (temporarily) lose some data than have a long outage so a
>> fully automated recovery is what I'd ideally like irrespective of
>> corruption.
>>
>> Cheers
>> Joe
>>
>>
>> JoeC wrote:
>>>
>>> I've upgraded to 5.4.2 and will let you know how it goes.
>>> I didn't rebuild the index as I've already restarted the process.
>>> In normal operation the queues should be empty for our application so
>>> that was not an issue for me.
>>>
>>> Thanks
>>> Joe
>>>
>>> On 23 February 2011 18:06, Gary Tully &lt;gary.tully@gmail.com&gt; wrote:
>>>> 5.4.2 is better w.r.t abortive shutdown, but for this case, rebuilding
>>>> the index should work.
>>>> remove kahadb/db.data and restart, it will parse the journal to
>>>> rebuild the index.
>>>>
>>>
>>
>>
>> --
>> View this message in context: http://activemq.2283324.n4.nabble.com/KahaDB-corruption-tp3321382p3695392.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
> --
> http://fusesource.com
> http://blog.garytully.com
>

Mime
View raw message