hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Date Tue, 14 Jul 2015 22:53:44 GMT
Start a vote thread? This says DISCUSSION

On Tue, Jul 14, 2015 at 2:05 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> There have been several iterations since the first mega patch was posted.
>
> Currently QA run is green.
>
> IntegrationTestIngestWithMOB has been run which passes.
>
> I want to give +1 for merging to master branch.
>
> On Tue, Jul 7, 2015 at 9:37 PM, Anoop John <anoop.hbase@gmail.com> wrote:
>
> > +1
> >
> > We will work on making it a Vote thread.
> >
> > -Anoop-
> >
> > On Wed, Jul 8, 2015 at 9:46 AM, ramkrishna vasudevan <
> > ramkrishna.s.vasudevan@gmail.com> wrote:
> >
> > > +1 to start the voting again.
> > >
> > > On Wed, Jul 8, 2015 at 9:07 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> > >
> > > > Looks like all MOB-related JIRAs have been resolved.
> > > >
> > > > Should the voting process be resumed ?
> > > >
> > > > Cheers
> > > >
> > > > On Thu, May 28, 2015 at 5:48 PM, Anoop John <anoop.hbase@gmail.com>
> > > wrote:
> > > >
> > > > > Yes Andy. The sweep tool is completely optional now. We have a
> chore
> > > > doing
> > > > > the compaction, like we trigger auto major compaction.  We can
> > > configure
> > > > > the interval. Auto can be turned off and user can explicitly call
> > also.
> > > > We
> > > > > have shell and API support.
> > > > >
> > > > > Anoop
> > > > >
> > > > > On Friday, May 29, 2015, Andrew Purtell <apurtell@apache.org>
> wrote:
> > > > > > MOB references in cells won't find their value if the MOB hfile
> has
> > > > been
> > > > > > corrupted. Dealing with that would be like any other corrupted
> > HFile,
> > > > > > understood. The dangling references make thinking about (partial)
> > > > > recovery
> > > > > > and repair interesting.
> > > > > >
> > > > > >
> > > > > > On Thursday, May 28, 2015, Jingcheng Du <jingcheng.du@intel.com>
> > > > wrote:
> > > > > >
> > > > > >> Andrew Purtell wrote
> > > > > >> > HBCK can check and sideline dangling reference files.
I think
> of
> > > MOB
> > > > > >> files
> > > > > >> > as "core enough" auxiliary files that need some support.
I
> > suppose
> > > > > unlike
> > > > > >> > reference files their presence or absence won't produce
a
> region
> > > > open
> > > > > >> > failure, we would see dangling pointers later when
tying to
> > > service
> > > > > >> > queries. (Yes?) Will that abort the RS? Pardon the
ignorant
> > > > question,
> > > > > >> > normally I could check the code but I'm at the airport
on a
> > phone.
> > > > > >>
> > > > > >> Thanks for comments!
> > > > > >> In mob, usually the reference cells are committed after
the mob
> > > files
> > > > > are
> > > > > >> done. I think it hardly happens that a reference cell cannot
> find
> > > its
> > > > > mob
> > > > > >> file.
> > > > > >> Even if there's a dangling reference cell, the RS won't
be
> > aborted,
> > > a
> > > > > empty
> > > > > >> cell is returned instead.
> > > > > >>
> > > > > >>
> > > > > >> Andrew Purtell wrote
> > > > > >> > On that subject, I should file follow up issues for
more check
> > and
> > > > > repair
> > > > > >> > options for HFiles. We should be able to detect missing
or
> > corrupt
> > > > > files
> > > > > >> > of
> > > > > >> > all variety: HFile, reference, MOB. This may require
an
> > expensive
> > > > scan
> > > > > >> > over
> > > > > >> > lots of files, but this is like fsck full disk surface
scans
> and
> > > > those
> > > > > >> > have
> > > > > >> > similar costs. Providing MR based tools is fine but
we should
> > have
> > > > > >> > multithreaded tools that can stand in if a MR runtime
is not
> > > > > available.
> > > > > >> > Import, Export, VerifyReplication...all of these tools
are in
> a
> > > > > >> different,
> > > > > >> > lesser, class than integrity and repair tools, in my
opinion.
> > > Since
> > > > > MOB
> > > > > >> > will likely be merged into trunk by then I'll be sure
to
> include
> > > > it. I
> > > > > >> > agree it's not fair to ask more of MOB then what we
have now
> for
> > > > > HFile.
> > > > > >>
> > > > > >> To detect the corrupt files, some code are needed in the
file
> > > checker
> > > > to
> > > > > >> check mob files after knowing it's a mob-enabled column.
> > > > > >> To detect the missing or dangling reference cells, I think
we
> have
> > > to
> > > > do
> > > > > a
> > > > > >> full-table scan, like what is done now in HFile.main.
> > > > > >> We can do that.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> View this message in context:
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
> > > > > >> Sent from the HBase Developer mailing list archive at
> Nabble.com.
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best regards,
> > > > > >
> > > > > >    - Andy
> > > > > >
> > > > > > Problems worthy of attack prove their worth by hitting back.
-
> Piet
> > > > Hein
> > > > > > (via Tom White)
> > > > > >
> > > > >
> > > >
> > >
> >
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message