hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Hsieh <...@cloudera.com>
Subject Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Date Tue, 14 Jul 2015 22:57:18 GMT
Since Jingcheng did the majority of the work, I'd like him to stat the
thread vote thread. It should happen in the next few days.

Jon

On Tue, Jul 14, 2015 at 3:53 PM, Andrew Purtell <apurtell@apache.org> wrote:

> Start a vote thread? This says DISCUSSION
>
> On Tue, Jul 14, 2015 at 2:05 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > There have been several iterations since the first mega patch was posted.
> >
> > Currently QA run is green.
> >
> > IntegrationTestIngestWithMOB has been run which passes.
> >
> > I want to give +1 for merging to master branch.
> >
> > On Tue, Jul 7, 2015 at 9:37 PM, Anoop John <anoop.hbase@gmail.com>
> wrote:
> >
> > > +1
> > >
> > > We will work on making it a Vote thread.
> > >
> > > -Anoop-
> > >
> > > On Wed, Jul 8, 2015 at 9:46 AM, ramkrishna vasudevan <
> > > ramkrishna.s.vasudevan@gmail.com> wrote:
> > >
> > > > +1 to start the voting again.
> > > >
> > > > On Wed, Jul 8, 2015 at 9:07 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> > > >
> > > > > Looks like all MOB-related JIRAs have been resolved.
> > > > >
> > > > > Should the voting process be resumed ?
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Thu, May 28, 2015 at 5:48 PM, Anoop John <anoop.hbase@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Yes Andy. The sweep tool is completely optional now. We have
a
> > chore
> > > > > doing
> > > > > > the compaction, like we trigger auto major compaction.  We can
> > > > configure
> > > > > > the interval. Auto can be turned off and user can explicitly
call
> > > also.
> > > > > We
> > > > > > have shell and API support.
> > > > > >
> > > > > > Anoop
> > > > > >
> > > > > > On Friday, May 29, 2015, Andrew Purtell <apurtell@apache.org>
> > wrote:
> > > > > > > MOB references in cells won't find their value if the MOB
hfile
> > has
> > > > > been
> > > > > > > corrupted. Dealing with that would be like any other corrupted
> > > HFile,
> > > > > > > understood. The dangling references make thinking about
> (partial)
> > > > > > recovery
> > > > > > > and repair interesting.
> > > > > > >
> > > > > > >
> > > > > > > On Thursday, May 28, 2015, Jingcheng Du <
> jingcheng.du@intel.com>
> > > > > wrote:
> > > > > > >
> > > > > > >> Andrew Purtell wrote
> > > > > > >> > HBCK can check and sideline dangling reference
files. I
> think
> > of
> > > > MOB
> > > > > > >> files
> > > > > > >> > as "core enough" auxiliary files that need some
support. I
> > > suppose
> > > > > > unlike
> > > > > > >> > reference files their presence or absence won't
produce a
> > region
> > > > > open
> > > > > > >> > failure, we would see dangling pointers later
when tying to
> > > > service
> > > > > > >> > queries. (Yes?) Will that abort the RS? Pardon
the ignorant
> > > > > question,
> > > > > > >> > normally I could check the code but I'm at the
airport on a
> > > phone.
> > > > > > >>
> > > > > > >> Thanks for comments!
> > > > > > >> In mob, usually the reference cells are committed after
the
> mob
> > > > files
> > > > > > are
> > > > > > >> done. I think it hardly happens that a reference cell
cannot
> > find
> > > > its
> > > > > > mob
> > > > > > >> file.
> > > > > > >> Even if there's a dangling reference cell, the RS won't
be
> > > aborted,
> > > > a
> > > > > > empty
> > > > > > >> cell is returned instead.
> > > > > > >>
> > > > > > >>
> > > > > > >> Andrew Purtell wrote
> > > > > > >> > On that subject, I should file follow up issues
for more
> check
> > > and
> > > > > > repair
> > > > > > >> > options for HFiles. We should be able to detect
missing or
> > > corrupt
> > > > > > files
> > > > > > >> > of
> > > > > > >> > all variety: HFile, reference, MOB. This may require
an
> > > expensive
> > > > > scan
> > > > > > >> > over
> > > > > > >> > lots of files, but this is like fsck full disk
surface scans
> > and
> > > > > those
> > > > > > >> > have
> > > > > > >> > similar costs. Providing MR based tools is fine
but we
> should
> > > have
> > > > > > >> > multithreaded tools that can stand in if a MR
runtime is not
> > > > > > available.
> > > > > > >> > Import, Export, VerifyReplication...all of these
tools are
> in
> > a
> > > > > > >> different,
> > > > > > >> > lesser, class than integrity and repair tools,
in my
> opinion.
> > > > Since
> > > > > > MOB
> > > > > > >> > will likely be merged into trunk by then I'll
be sure to
> > include
> > > > > it. I
> > > > > > >> > agree it's not fair to ask more of MOB then what
we have now
> > for
> > > > > > HFile.
> > > > > > >>
> > > > > > >> To detect the corrupt files, some code are needed in
the file
> > > > checker
> > > > > to
> > > > > > >> check mob files after knowing it's a mob-enabled column.
> > > > > > >> To detect the missing or dangling reference cells,
I think we
> > have
> > > > to
> > > > > do
> > > > > > a
> > > > > > >> full-table scan, like what is done now in HFile.main.
> > > > > > >> We can do that.
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >> View this message in context:
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
> > > > > > >> Sent from the HBase Developer mailing list archive
at
> > Nabble.com.
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best regards,
> > > > > > >
> > > > > > >    - Andy
> > > > > > >
> > > > > > > Problems worthy of attack prove their worth by hitting
back. -
> > Piet
> > > > > Hein
> > > > > > > (via Tom White)
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>



-- 
// Jonathan Hsieh (shay)
// HBase Tech Lead, Software Engineer, Cloudera
// jon@cloudera.com // @jmhsieh

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message