Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CA+RK=_CdgSdS7uabOSfSGCbPuri0LYM_ogxe_Qtozu19_qakfw@mail.gmail.com>
References: 
 <CAAha9a2sRngY-i9vQUzE_4sQ3cNRK1e4Erxcj=LM3LmmEkhhxg@mail.gmail.com>
	<CALte62yLbrdtLH23+C4mQ7QW7Ot5HpCzoaFaEVp6wEit0zek+g@mail.gmail.com>
	<CAAha9a0n-vN7Bz-ixW=Dz-ku3QW+=yTwPC8h3tyXf6SFW-uz9w@mail.gmail.com>
	<CA+RK=_BNN6y8v1R4CcYgPNWviJz5kaC9oY7hmsgoCrU_8m8oMA@mail.gmail.com>
	<CA+RK=_AuCTh2uf1Ty3afm3ZF_SDuukiTccAiXzU_1h=74G--fQ@mail.gmail.com>
	<CAFb6HiL79U_TaEAKX3hDf+OSbM48ZKVnF62vpvGfLbPU+sruzQ@mail.gmail.com>
	<CA+RK=_AFju1Kr_q3--T5zu-EVbKeuanWs4B8TkuzTHWqzO1QoA@mail.gmail.com>
	<CAAha9a1BwU9_G-JVoChQw2xp1+ry6njr==HzPU1H_zt7hLOtag@mail.gmail.com>
	<CA+RK=_BBVmni4hx_fHaRShm3VQ0y-xGaM4Rb9q3sUymLv_yjNw@mail.gmail.com>
	<1432791830121-4071911.post@n3.nabble.com>
	<CA+RK=_CdgSdS7uabOSfSGCbPuri0LYM_ogxe_Qtozu19_qakfw@mail.gmail.com>
Date: Fri, 29 May 2015 06:18:34 +0530
Message-ID: 
 <CAOtJ30on7xQztTQyPw5WF8wijALKZ1AoOaaC+J6pRrX19vR42Q@mail.gmail.com>
Subject: Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
From: Anoop John <anoop.hbase@gmail.com>
To: "dev@hbase.apache.org" <dev@hbase.apache.org>
Cc: "hbase-dev@hadoop.apache.org" <hbase-dev@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=089e0160a330fb6ba905172dd5ba

--089e0160a330fb6ba905172dd5ba
Content-Type: text/plain; charset=UTF-8

Yes Andy. The sweep tool is completely optional now. We have a chore doing
the compaction, like we trigger auto major compaction.  We can configure
the interval. Auto can be turned off and user can explicitly call also. We
have shell and API support.

Anoop

On Friday, May 29, 2015, Andrew Purtell <apurtell@apache.org> wrote:
> MOB references in cells won't find their value if the MOB hfile has been
> corrupted. Dealing with that would be like any other corrupted HFile,
> understood. The dangling references make thinking about (partial) recovery
> and repair interesting.
>
>
> On Thursday, May 28, 2015, Jingcheng Du <jingcheng.du@intel.com> wrote:
>
>> Andrew Purtell wrote
>> > HBCK can check and sideline dangling reference files. I think of MOB
>> files
>> > as "core enough" auxiliary files that need some support. I suppose
unlike
>> > reference files their presence or absence won't produce a region open
>> > failure, we would see dangling pointers later when tying to service
>> > queries. (Yes?) Will that abort the RS? Pardon the ignorant question,
>> > normally I could check the code but I'm at the airport on a phone.
>>
>> Thanks for comments!
>> In mob, usually the reference cells are committed after the mob files are
>> done. I think it hardly happens that a reference cell cannot find its mob
>> file.
>> Even if there's a dangling reference cell, the RS won't be aborted, a
empty
>> cell is returned instead.
>>
>>
>> Andrew Purtell wrote
>> > On that subject, I should file follow up issues for more check and
repair
>> > options for HFiles. We should be able to detect missing or corrupt
files
>> > of
>> > all variety: HFile, reference, MOB. This may require an expensive scan
>> > over
>> > lots of files, but this is like fsck full disk surface scans and those
>> > have
>> > similar costs. Providing MR based tools is fine but we should have
>> > multithreaded tools that can stand in if a MR runtime is not available.
>> > Import, Export, VerifyReplication...all of these tools are in a
>> different,
>> > lesser, class than integrity and repair tools, in my opinion. Since MOB
>> > will likely be merged into trunk by then I'll be sure to include it. I
>> > agree it's not fair to ask more of MOB then what we have now for HFile.
>>
>> To detect the corrupt files, some code are needed in the file checker to
>> check mob files after knowing it's a mob-enabled column.
>> To detect the missing or dangling reference cells, I think we have to do
a
>> full-table scan, like what is done now in HFile.main.
>> We can do that.
>>
>>
>>
>> --
>> View this message in context:
>>
http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
>> Sent from the HBase Developer mailing list archive at Nabble.com.
>>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

--089e0160a330fb6ba905172dd5ba--