Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of chilinglam@gmail.com
 designates 209.85.214.169 as permitted sender)
References: 
 <CANZDn9tg5v08Dxhbtu4+0COs6D_ss2d=9e8wpYQgqvpO2ioQEg@mail.gmail.com>
 <CADcMMgGxNpH9yEV6GLBubqfYpP4d=Qg6syh4PL=f88Nf60u73Q@mail.gmail.com>
 <CANZDn9uFiaOV_iG5=ZUfQ2zBB8tMzyjp-zDROE-g2E0NYL+txA@mail.gmail.com>
 <CAGWW7L=ND9_cQkOWU1Gs0zA7EQLKARUz91VnSW0gM=xp9aSHsQ@mail.gmail.com>
 <CANZDn9u9q0_MH499vbnJKULcxfi9N_aRGF73ktGejhs6mm4Acw@mail.gmail.com>
 <CAGWW7LnWiD7+eaThUb_cUDr1xnASRNtbosi=wjHTvu4mKnNj=w@mail.gmail.com>
In-Reply-To: 
 <CAGWW7LnWiD7+eaThUb_cUDr1xnASRNtbosi=wjHTvu4mKnNj=w@mail.gmail.com>
Mime-Version: 1.0 (1.0)
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii
Message-Id: <9B9DCFA6-BA17-496C-B475-432C6D47684D@gmail.com>
Cc: "user@hbase.apache.org" <user@hbase.apache.org>
From: Jerry Lam <chilinglam@gmail.com>
Subject: Re: Recovering corrupt HLog files
Date: Sat, 30 Jun 2012 21:55:00 -0400
To: "user@hbase.apache.org" <user@hbase.apache.org>

This is interesting because I saw this happens in the past. Is walplayer can=
 be back ported to 0.90.x?=20

Best Regards,

Jerry=20

Sent from my iPad

On 2012-06-30, at 16:34, Li Pi <li@idle.li> wrote:

> Nope. It came out in 0.94 otoh.
>=20
> On Sat, Jun 30, 2012 at 12:29 PM, Bryan Beaudreault <
> bbeaudreault@hubspot.com> wrote:
>=20
>> I should have mentioned in my initial email that I am operating on HBase
>> 0.90.4.  Is WALPlayer available in this version?  I am having trouble
>> finding it or anything similar.
>>=20
>> On Sat, Jun 30, 2012 at 1:14 PM, Li Pi <li@idle.li> wrote:
>>=20
>>> WALPlayer will look at the timestamp. Replaying an older edit that has
>>> since been overwritten shouldn't change anything.
>>>=20
>>> On Sat, Jun 30, 2012 at 9:49 AM, Bryan Beaudreault <
>>> bbeaudreault@hubspot.com
>>>> wrote:
>>>=20
>>>> They are all pretty large, around 40+mb.  Will the walplayer be smart
>>>> enough to only write edits that still look relevant (i.e. based on
>>>> timestamps of the edits vs timestamps of the versions in hbase)?
>> Writes
>>>> have been coming in since we recovered.
>>>>=20
>>>> On Sat, Jun 30, 2012 at 11:05 AM, Stack <stack@duboce.net> wrote:
>>>>=20
>>>>> On Sat, Jun 30, 2012 at 8:38 AM, Bryan Beaudreault
>>>>> <bbeaudreault@hubspot.com> wrote:
>>>>>> 12/06/30 00:00:48 INFO wal.HLogSplitter: Got while parsing hlog
>>>>>>=20
>>>>>=20
>>>>=20
>>>=20
>> hdfs://my-namenode-ip-addr:8020/hbase/.logs/my-rs-ip-addr,60020,133866771=
9591/my-rs-ip-addr%3A60020.1340935453874.
>>>>>> Marking as corrupted
>>>>>>=20
>>>>>=20
>>>>> What size do these logs have?
>>>>>=20
>>>>>> We are back to stable operating now, and in trying to research
>> this I
>>>>> found
>>>>>> the hdfs://my-namenode-ip-addr:8020/hbase/.corrupt directory.
>> There
>>>> are
>>>>> 20
>>>>>> files listed there.
>>>>>>=20
>>>>>=20
>>>>> Ditto.
>>>>>=20
>>>>>> What are our options for tracking down and potentially recovering
>> any
>>>>> data
>>>>>> that was lost.  Or how can we even tell what was lost, if any?
>> Does
>>>> the
>>>>>> existence of these files pretty much guarantee data lost? There
>>> doesn't
>>>>>> seem to be much documentation on this.  =46rom reading it seems like
>> it
>>>>> might
>>>>>> be possible that part of each of these files was recovered.
>>>>>>=20
>>>>>=20
>>>>> If size > 0, could try walplaying them:
>>>>> http://hbase.apache.org/book.html#walplayer
>>>>>=20
>>>>> St.Ack
>>>>>=20
>>>>=20
>>>=20
>>=20