Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CA+RK=_Ang+g+eFY3WLhPTxbZ7pGfPhEOJouHAJEjHXq4mKBn9A@mail.gmail.com>
References: 
 <CAAha9a2sRngY-i9vQUzE_4sQ3cNRK1e4Erxcj=LM3LmmEkhhxg@mail.gmail.com>
	<CALte62yLbrdtLH23+C4mQ7QW7Ot5HpCzoaFaEVp6wEit0zek+g@mail.gmail.com>
	<CAAha9a0n-vN7Bz-ixW=Dz-ku3QW+=yTwPC8h3tyXf6SFW-uz9w@mail.gmail.com>
	<CA+RK=_BNN6y8v1R4CcYgPNWviJz5kaC9oY7hmsgoCrU_8m8oMA@mail.gmail.com>
	<CA+RK=_AuCTh2uf1Ty3afm3ZF_SDuukiTccAiXzU_1h=74G--fQ@mail.gmail.com>
	<CAFb6HiL79U_TaEAKX3hDf+OSbM48ZKVnF62vpvGfLbPU+sruzQ@mail.gmail.com>
	<CA+RK=_AFju1Kr_q3--T5zu-EVbKeuanWs4B8TkuzTHWqzO1QoA@mail.gmail.com>
	<CAAha9a1BwU9_G-JVoChQw2xp1+ry6njr==HzPU1H_zt7hLOtag@mail.gmail.com>
	<CA+RK=_BBVmni4hx_fHaRShm3VQ0y-xGaM4Rb9q3sUymLv_yjNw@mail.gmail.com>
	<1432791830121-4071911.post@n3.nabble.com>
	<CA+RK=_CdgSdS7uabOSfSGCbPuri0LYM_ogxe_Qtozu19_qakfw@mail.gmail.com>
	<CAOtJ30on7xQztTQyPw5WF8wijALKZ1AoOaaC+J6pRrX19vR42Q@mail.gmail.com>
	<CALte62xcAdc1QfkSNzPUoq7Z4-KpyAw6_RsLa=UBG3Bzk17DdA@mail.gmail.com>
	<CAAT7MkrqQfyKnXJwKnyZR_sUHxNFTCtBMFsTBAC3GVNf1FRO6A@mail.gmail.com>
	<CAOtJ30qm8M7cZ_xPVWmZSyqueoQ7xwVKVs0AsFeCrWhzJMwYwA@mail.gmail.com>
	<CALte62w6i98ELd9AypHtJ-=MU8DQbS9ozaz2QDEdPQuVSMKPXA@mail.gmail.com>
	<CA+RK=_DG42smbMaB8X=TnREu5_4+oDti3gwRPv4FZX-nBgEZuA@mail.gmail.com>
	<CAAha9a2mquBskcH=+wCDwy5f8pRYEJRsyUnXvUQ7CjbCBmZNUw@mail.gmail.com>
	<CA+RK=_Ang+g+eFY3WLhPTxbZ7pGfPhEOJouHAJEjHXq4mKBn9A@mail.gmail.com>
Date: Thu, 16 Jul 2015 20:41:35 -0700
Message-ID: 
 <CALte62wXFNkk25CLdZ6EC0vtw=caTBf9tW2W5X-Vt39jH6BrUA@mail.gmail.com>
Subject: Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
From: Ted Yu <yuzhihong@gmail.com>
To: "dev@hbase.apache.org" <dev@hbase.apache.org>, "Du,
 Jingcheng" <jingcheng.du@intel.com>
Cc: "hbase-dev@hadoop.apache.org" <hbase-dev@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=001a113a6cb6f953a3051b09f6de

--001a113a6cb6f953a3051b09f6de
Content-Type: text/plain; charset=UTF-8

Jingcheng:
Let us know if you need anything for composition of the vote thread.

On Tue, Jul 14, 2015 at 3:58 PM, Andrew Purtell <apurtell@apache.org> wrote:

> That's great, and I also appreciate the time spent discussing on this
> thread. I plan to vote yes.
>
> On Tue, Jul 14, 2015 at 3:57 PM, Jonathan Hsieh <jon@cloudera.com> wrote:
>
> > Since Jingcheng did the majority of the work, I'd like him to stat the
> > thread vote thread. It should happen in the next few days.
> >
> > Jon
> >
> > On Tue, Jul 14, 2015 at 3:53 PM, Andrew Purtell <apurtell@apache.org>
> > wrote:
> >
> > > Start a vote thread? This says DISCUSSION
> > >
> > > On Tue, Jul 14, 2015 at 2:05 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > >
> > > > There have been several iterations since the first mega patch was
> > posted.
> > > >
> > > > Currently QA run is green.
> > > >
> > > > IntegrationTestIngestWithMOB has been run which passes.
> > > >
> > > > I want to give +1 for merging to master branch.
> > > >
> > > > On Tue, Jul 7, 2015 at 9:37 PM, Anoop John <anoop.hbase@gmail.com>
> > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > We will work on making it a Vote thread.
> > > > >
> > > > > -Anoop-
> > > > >
> > > > > On Wed, Jul 8, 2015 at 9:46 AM, ramkrishna vasudevan <
> > > > > ramkrishna.s.vasudevan@gmail.com> wrote:
> > > > >
> > > > > > +1 to start the voting again.
> > > > > >
> > > > > > On Wed, Jul 8, 2015 at 9:07 AM, Ted Yu <yuzhihong@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Looks like all MOB-related JIRAs have been resolved.
> > > > > > >
> > > > > > > Should the voting process be resumed ?
> > > > > > >
> > > > > > > Cheers
> > > > > > >
> > > > > > > On Thu, May 28, 2015 at 5:48 PM, Anoop John <
> > anoop.hbase@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Yes Andy. The sweep tool is completely optional now. We have
> a
> > > > chore
> > > > > > > doing
> > > > > > > > the compaction, like we trigger auto major compaction.  We
> can
> > > > > > configure
> > > > > > > > the interval. Auto can be turned off and user can explicitly
> > call
> > > > > also.
> > > > > > > We
> > > > > > > > have shell and API support.
> > > > > > > >
> > > > > > > > Anoop
> > > > > > > >
> > > > > > > > On Friday, May 29, 2015, Andrew Purtell <apurtell@apache.org
> >
> > > > wrote:
> > > > > > > > > MOB references in cells won't find their value if the MOB
> > hfile
> > > > has
> > > > > > > been
> > > > > > > > > corrupted. Dealing with that would be like any other
> > corrupted
> > > > > HFile,
> > > > > > > > > understood. The dangling references make thinking about
> > > (partial)
> > > > > > > > recovery
> > > > > > > > > and repair interesting.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thursday, May 28, 2015, Jingcheng Du <
> > > jingcheng.du@intel.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Andrew Purtell wrote
> > > > > > > > >> > HBCK can check and sideline dangling reference files. I
> > > think
> > > > of
> > > > > > MOB
> > > > > > > > >> files
> > > > > > > > >> > as "core enough" auxiliary files that need some
> support. I
> > > > > suppose
> > > > > > > > unlike
> > > > > > > > >> > reference files their presence or absence won't produce
> a
> > > > region
> > > > > > > open
> > > > > > > > >> > failure, we would see dangling pointers later when tying
> > to
> > > > > > service
> > > > > > > > >> > queries. (Yes?) Will that abort the RS? Pardon the
> > ignorant
> > > > > > > question,
> > > > > > > > >> > normally I could check the code but I'm at the airport
> on
> > a
> > > > > phone.
> > > > > > > > >>
> > > > > > > > >> Thanks for comments!
> > > > > > > > >> In mob, usually the reference cells are committed after
> the
> > > mob
> > > > > > files
> > > > > > > > are
> > > > > > > > >> done. I think it hardly happens that a reference cell
> cannot
> > > > find
> > > > > > its
> > > > > > > > mob
> > > > > > > > >> file.
> > > > > > > > >> Even if there's a dangling reference cell, the RS won't be
> > > > > aborted,
> > > > > > a
> > > > > > > > empty
> > > > > > > > >> cell is returned instead.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> Andrew Purtell wrote
> > > > > > > > >> > On that subject, I should file follow up issues for more
> > > check
> > > > > and
> > > > > > > > repair
> > > > > > > > >> > options for HFiles. We should be able to detect missing
> or
> > > > > corrupt
> > > > > > > > files
> > > > > > > > >> > of
> > > > > > > > >> > all variety: HFile, reference, MOB. This may require an
> > > > > expensive
> > > > > > > scan
> > > > > > > > >> > over
> > > > > > > > >> > lots of files, but this is like fsck full disk surface
> > scans
> > > > and
> > > > > > > those
> > > > > > > > >> > have
> > > > > > > > >> > similar costs. Providing MR based tools is fine but we
> > > should
> > > > > have
> > > > > > > > >> > multithreaded tools that can stand in if a MR runtime is
> > not
> > > > > > > > available.
> > > > > > > > >> > Import, Export, VerifyReplication...all of these tools
> are
> > > in
> > > > a
> > > > > > > > >> different,
> > > > > > > > >> > lesser, class than integrity and repair tools, in my
> > > opinion.
> > > > > > Since
> > > > > > > > MOB
> > > > > > > > >> > will likely be merged into trunk by then I'll be sure to
> > > > include
> > > > > > > it. I
> > > > > > > > >> > agree it's not fair to ask more of MOB then what we have
> > now
> > > > for
> > > > > > > > HFile.
> > > > > > > > >>
> > > > > > > > >> To detect the corrupt files, some code are needed in the
> > file
> > > > > > checker
> > > > > > > to
> > > > > > > > >> check mob files after knowing it's a mob-enabled column.
> > > > > > > > >> To detect the missing or dangling reference cells, I think
> > we
> > > > have
> > > > > > to
> > > > > > > do
> > > > > > > > a
> > > > > > > > >> full-table scan, like what is done now in HFile.main.
> > > > > > > > >> We can do that.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> --
> > > > > > > > >> View this message in context:
> > > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
> > > > > > > > >> Sent from the HBase Developer mailing list archive at
> > > > Nabble.com.
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Best regards,
> > > > > > > > >
> > > > > > > > >    - Andy
> > > > > > > > >
> > > > > > > > > Problems worthy of attack prove their worth by hitting
> back.
> > -
> > > > Piet
> > > > > > > Hein
> > > > > > > > > (via Tom White)
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > >    - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
> >
> >
> > --
> > // Jonathan Hsieh (shay)
> > // HBase Tech Lead, Software Engineer, Cloudera
> > // jon@cloudera.com // @jmhsieh
> >
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

--001a113a6cb6f953a3051b09f6de--