hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Date Thu, 28 May 2015 22:51:06 GMT
MOB references in cells won't find their value if the MOB hfile has been
corrupted. Dealing with that would be like any other corrupted HFile,
understood. The dangling references make thinking about (partial) recovery
and repair interesting.

On Thursday, May 28, 2015, Jingcheng Du <jingcheng.du@intel.com> wrote:

> Andrew Purtell wrote
> > HBCK can check and sideline dangling reference files. I think of MOB
> files
> > as "core enough" auxiliary files that need some support. I suppose unlike
> > reference files their presence or absence won't produce a region open
> > failure, we would see dangling pointers later when tying to service
> > queries. (Yes?) Will that abort the RS? Pardon the ignorant question,
> > normally I could check the code but I'm at the airport on a phone.
> Thanks for comments!
> In mob, usually the reference cells are committed after the mob files are
> done. I think it hardly happens that a reference cell cannot find its mob
> file.
> Even if there's a dangling reference cell, the RS won't be aborted, a empty
> cell is returned instead.
> Andrew Purtell wrote
> > On that subject, I should file follow up issues for more check and repair
> > options for HFiles. We should be able to detect missing or corrupt files
> > of
> > all variety: HFile, reference, MOB. This may require an expensive scan
> > over
> > lots of files, but this is like fsck full disk surface scans and those
> > have
> > similar costs. Providing MR based tools is fine but we should have
> > multithreaded tools that can stand in if a MR runtime is not available.
> > Import, Export, VerifyReplication...all of these tools are in a
> different,
> > lesser, class than integrity and repair tools, in my opinion. Since MOB
> > will likely be merged into trunk by then I'll be sure to include it. I
> > agree it's not fair to ask more of MOB then what we have now for HFile.
> To detect the corrupt files, some code are needed in the file checker to
> check mob files after knowing it's a mob-enabled column.
> To detect the missing or dangling reference cells, I think we have to do a
> full-table scan, like what is done now in HFile.main.
> We can do that.
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
> Sent from the HBase Developer mailing list archive at Nabble.com.

Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message