Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of ramkrishna.vasudevan@huawei.com
 designates 119.145.14.67 as permitted sender)
Date: Wed, 25 Jan 2012 11:57:33 +0530
From: "Ramkrishna.S.Vasudevan" <ramkrishna.vasudevan@huawei.com>
Subject: RE: CatalogJanitor trying to delete an already deleted region from
 META
In-reply-to: 
 <CADcMMgGBzrEBhOONxQeAWLAJTbD-tgeRGFk=05F9SLGR9AWV2g@mail.gmail.com>
To: dev@hbase.apache.org
Message-id: <005e01ccdb2a$6e56e680$4b04b380$%vasudevan@huawei.com>
MIME-version: 1.0
Content-type: text/plain; charset=iso-8859-1
Content-language: en-us
Content-transfer-encoding: quoted-printable
Thread-index: AczbJ10n5curlTqBR1uc2C01KpX2WwAAuJkg
References: <004001ccdb1c$15958730$40c09590$%vasudevan@huawei.com>
 <CADcMMgGBzrEBhOONxQeAWLAJTbD-tgeRGFk=05F9SLGR9AWV2g@mail.gmail.com>

Hi Stack
Thanks for your reply.  As per the logs not many Store files for META =
only 1
HFile for META.  We are carrying out different tests.  Any ideas are
welcome!!

Regards
Ram

-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of =
Stack
Sent: Wednesday, January 25, 2012 11:35 AM
To: dev@hbase.apache.org
Subject: Re: CatalogJanitor trying to delete an already deleted region =
from
META

On Tue, Jan 24, 2012 at 8:44 PM, Ramkrishna.S.Vasudevan
<ramkrishna.vasudevan@huawei.com> wrote:
> The region 9e28c97ccec104f3174ecf343cb59157 is a parent region. =A0It =
got
> splitted into 2 daughter regions.
>
> The CatalogJanitor deleted this entry from META. =A0Subsequent run of
Catalog
> Janitor did not find the deleted region from META. =A0After 5 times of
> CatalogJanitor running, it again found the deleted region.
>

Thats odd Ram.


> Anyway this time as the region was deleted from HDFS we got the below =
warn
> log.
>
>
> 2012-01-13 07:46:49,552 WARN =
org.apache.hadoop.hbase.regionserver.HRegion:
> Failed delete of
>
hdfs://193.195.18.121:9000/hbase/Htable_UFDR_037/9e28c97ccec104f3174ecf34=
3cb
> 59157
>
> But why did the CatalogJanitor again pick up this deleted region in =
META.
> fullScan(). =A0The same scenario has happened for about 6 regions in =
the
same
> time. =A0From the RS logs we could not find any operation pertaining =
to this
> region during that time.
>
> Any clues or suggestions?
>

In old days, back before 0.20 IIRC, we had an issue where edits could
resurface in .META. because we were doing the merge of deletes and
edits from different store files 'wrong'.  A full-compaction would
'fix it'.   Were there many storefiles in .META. at the time?  Maybe
something similar (You'd have thought we'd have seen this more
frequently if it were the case that we still had this 'bug').

St.Ack