hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Hsieh <...@cloudera.com>
Subject Re: MOB in branch-1? (Re: [RESULT][VOTE] Merge branch hbase-11339 HBase MOB to trunk)
Date Tue, 01 Mar 2016 02:56:05 GMT
The feature is definitely not abandoned -- we have few sizable customers at
PB scale that I can recall off the top of my head that have been using it
for over 8-12 months in the version backported to CDH (we backported an
early version back in oct/14 (CDH5.2), and updated with the recent more
recent changes in roughly may/15 (CDH5.4).   These are used primarily to
store documents -- think PDF files. They are pretty happy -- the customer
reported that it was slightly slower on the write side (10-15%) than a
competing system but significantly faster on the read side (3x-4x
throughput).

Over the holidays we shook out a semi-rare data loss issue with mob
compaction that had this as the root cause[1] -- (since pointers are only
stored in the "normal hfiles" the volume of datas on this case was fairly
large).

We've been working on trying to get at least one of them to present at
hbasecon, but I'm not reviewing submissions this year and don't know if
they made it or not.

Jon.

[1] https://issues.apache.org/jira/browse/HBASE-15035

On Fri, Feb 26, 2016 at 12:27 PM, Andrew Purtell <apurtell@apache.org>
wrote:

> ​I think we need at least one success story or one very interested user
> with a real project on the line to justify a backport. Otherwise it's a
> feature without any users - technically, abandoned. ​
>
> On Fri, Feb 26, 2016 at 10:11 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > I am interested in hearing about user experience with MOB feature as
> well.
> >
> > In my opinion, this feature is a nice addition to branch-1.
> >
> > On Wed, Feb 10, 2016 at 1:45 PM, Andrew Purtell <apurtell@apache.org>
> > wrote:
> >
> > > +user@
> > >
> > > Is there anyone using the MOB feature in trunk for anything who can
> > comment
> > > on how well it's been working out? Intel folks maybe?
> > >
> > > On Thu, Jan 21, 2016 at 7:46 AM, Sean Busbey <busbey@apache.org>
> wrote:
> > >
> > > > The last time MOB on branch-1 came up, folks were concerned that it
> > > > wasn't stable enough in master yet. Is that still the case?
> > > >
> > > > Can we get a [DISCUSS] flagged thread to see what, if anything, folks
> > > > would like to see gate inclusion in branch-1?
> > > >
> > > > On Tue, Jan 19, 2016 at 7:31 PM, Jonathan Hsieh <jon@cloudera.com>
> > > wrote:
> > > > > +1 to 1.2 being feature complete corrently.  There has already
> been a
> > > > > release candidate and folks are burning down the blockers currently
> > to
> > > > prep
> > > > > for the next RC.
> > > > >
> > > > > I like the idea of mob and sparkonhbase for 1.3.  I'm more
> > comfortable
> > > > with
> > > > > sparkonhbase -- it is a new module and thus not as invasive.
> > > > >
> > > > > Jon.
> > > > >
> > > > > On Tue, Jan 19, 2016 at 4:55 PM, Andrew Purtell <
> > > > andrew.purtell@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Pretty sure Sean expressed 1.2 is feature complete and I'd support
> > > that.
> > > > >> Can we wait for 1.3 for MOB ? Can look at Spark connector then
> too.
> > > > >>
> > > > >> > On Jan 19, 2016, at 4:52 PM, Ted Yu <yuzhihong@gmail.com>
> wrote:
> > > > >> >
> > > > >> > Looks like 1.2.0 RC is in near future.
> > > > >> >
> > > > >> > I wonder if it is time to revive this thread (due to customer
> > > > interest).
> > > > >> >
> > > > >> > As far as I can tell, the Mob related tests have been passing
in
> > the
> > > > >> recent
> > > > >> > past.
> > > > >> >
> > > > >> > Thanks
> > > > >> >
> > > > >> >> On Wed, Oct 28, 2015 at 2:07 PM, Andrew Purtell <
> > > apurtell@apache.org
> > > > >
> > > > >> wrote:
> > > > >> >>
> > > > >> >> I haven't heard an user answer in the affirmative to
wanting
> it.
> > > > >> >>
> > > > >> >> I'll volunteer to RM 1.3, whenever we need it. Premature
to
> have
> > > that
> > > > >> >> discussion without 1.2 even out the door yet, though.
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >> On Wed, Oct 28, 2015 at 1:01 PM, Stephen Jiang <
> > > > syuanjiangdev@gmail.com
> > > > >> >
> > > > >> >> wrote:
> > > > >> >>
> > > > >> >>> Actually, it is actively changing in master branch
on MOB
> > feature
> > > > made
> > > > >> me
> > > > >> >>> think about: if we ever want to port MOB feature
to branch-1,
> > now
> > > > is a
> > > > >> >> good
> > > > >> >>> time.  We can commit changes in both branches; otherwise,
we
> > > > probably
> > > > >> >> would
> > > > >> >>> miss some commits when we port MOB to branch-1 in
a late time.
> > > > >> >>>
> > > > >> >>> I am more thinking about 1.3 release (certainly
not 1.2),
> which
> > we
> > > > >> still
> > > > >> >>> have some time to stabilize and allow interesting
party to
> play
> > > with
> > > > >> the
> > > > >> >>> feature and give feedback.
> > > > >> >>>
> > > > >> >>> Thanks
> > > > >> >>> Stephen
> > > > >> >>>
> > > > >> >>> PS. given the features we discussed in 2.0.0 in
the last
> > community
> > > > >> >> meeting,
> > > > >> >>> I think it would not release earlier than 1.3 :-),
unless we
> > > > >> >> intentionally
> > > > >> >>> not find a release manager for 1.3.
> > > > >> >>>
> > > > >> >>>
> > > > >> >>>
> > > > >> >>>
> > > > >> >>>> On Wed, Oct 28, 2015 at 9:09 AM, Sean Busbey
<
> > > busbey@cloudera.com>
> > > > >> >>> wrote:
> > > > >> >>>
> > > > >> >>>> It's practically November. Matteo, are you up
for a thread on
> > > > target
> > > > >> >>>> dates for 2.0.0 to start RCs?
> > > > >> >>>>
> > > > >> >>>>
> > > > >> >>>> On Wed, Oct 28, 2015 at 11:02 AM, Elliott Clark
<
> > > eclark@apache.org
> > > > >
> > > > >> >>> wrote:
> > > > >> >>>>> I feel the same lets keep branch-1 stable,
and work towards
> a
> > > > faster
> > > > >> >>>> 2.0.0.
> > > > >> >>>>>
> > > > >> >>>>>> On Tue, Oct 27, 2015 at 4:52 PM, Stack
<stack@duboce.net>
> > > wrote:
> > > > >> >>>>>>
> > > > >> >>>>>> IMO, MOB is still not settled in Master.
It has a bunch of
> > > flakey
> > > > >> >>> tests
> > > > >> >>>>>> that are getting fixed by Jingcheng
or I've disabled them
> > till
> > > > >> >> someone
> > > > >> >>>> has
> > > > >> >>>>>> time to look at them. There is also
a load of duplicated
> code
> > > > that
> > > > >> >> is
> > > > >> >>>> being
> > > > >> >>>>>> cleaned up (Matteo).
> > > > >> >>>>>>
> > > > >> >>>>>> Its not ready to go back to branch-1
IMO.
> > > > >> >>>>>>
> > > > >> >>>>>> Are there users who'd like it backported?
> > > > >> >>>>>> St.Ack
> > > > >> >>>>>>
> > > > >> >>>>>>
> > > > >> >>>>>> On Mon, Oct 26, 2015 at 10:55 AM, Stephen
Jiang <
> > > > >> >>>> syuanjiangdev@gmail.com>
> > > > >> >>>>>> wrote:
> > > > >> >>>>>>
> > > > >> >>>>>>> Hello, guys, the MOB is in master
branch.  I saw bug fixes
> > > > >> >> happening
> > > > >> >>>> in
> > > > >> >>>>>>> master branch.
> > > > >> >>>>>>>
> > > > >> >>>>>>> I just wonder whether there is a
plan to put MOB in
> > > branch-1.  I
> > > > >> >> am
> > > > >> >>>>>> afraid
> > > > >> >>>>>>> if we don't do it now, it would
be harder in the future to
> > > back
> > > > >> >> port
> > > > >> >>>> if
> > > > >> >>>>>> we
> > > > >> >>>>>>> decide to do it in a late time.
> > > > >> >>>>>>>
> > > > >> >>>>>>> Thanks
> > > > >> >>>>>>> Stephen
> > > > >> >>>>>>>
> > > > >> >>>>>>> On Thu, Jul 23, 2015 at 1:15 PM,
Andrew Purtell <
> > > > >> >>> apurtell@apache.org>
> > > > >> >>>>>>> wrote:
> > > > >> >>>>>>>
> > > > >> >>>>>>>> Thanks Jon.
> > > > >> >>>>>>>>
> > > > >> >>>>>>>> When I'm back in the office
I'll check out master and
> have
> > a
> > > > >> >> look
> > > > >> >>>> into
> > > > >> >>>>>>> any
> > > > >> >>>>>>>> locally repeatable test failures.
Anyway in my opinion at
> > > this
> > > > >> >>>> point it
> > > > >> >>>>>>>> would make the most sense for
us to keep the MOB changes
> in
> > > on
> > > > >> >>>> master
> > > > >> >>>>>> and
> > > > >> >>>>>>>> deal with any fallout in follow
on issues. I think all
> who
> > > > voted
> > > > >> >>> +1
> > > > >> >>>> for
> > > > >> >>>>>>>> this change were aware that
large changes like this can
> > have
> > > a
> > > > >> >>>>>>> temporarily
> > > > >> >>>>>>>> destabilizing effect. As long
as the MOB devs are around
> to
> > > > help
> > > > >> >>>> clean
> > > > >> >>>>>>> up,
> > > > >> >>>>>>>> we should be good!
> > > > >> >>>>>>>>
> > > > >> >>>>>>>>
> > > > >> >>>>>>>> On Wed, Jul 22, 2015 at 8:09
PM, Jonathan Hsieh <
> > > > >> >> jon@cloudera.com
> > > > >> >>>>
> > > > >> >>>>>>> wrote:
> > > > >> >>>>>>>>
> > > > >> >>>>>>>>> I had two clean full builds/unit
test on my internal
> setup
> > > and
> > > > >> >>> the
> > > > >> >>>>>>> latest
> > > > >> >>>>>>>>> build went back to ~4325
total tests and failures on
> > > Procedure
> > > > >> >>>> relate
> > > > >> >>>>>>>> tests
> > > > >> >>>>>>>>> cases.
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>> I don't think mob is responsible
for these failures.
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>> Jon.
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>> On Wed, Jul 22, 2015 at
4:32 PM, Jonathan Hsieh <
> > > > >> >>> jon@cloudera.com
> > > > >> >>>>>
> > > > >> >>>>>>>> wrote:
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>>> Although the the precommit
buiid passed, and the
> > > compilation
> > > > >> >>> and
> > > > >> >>>>>> mob
> > > > >> >>>>>>>>>> testing I ran after
before the merge was commited
> passed,
> > > It
> > > > >> >>>> looks
> > > > >> >>>>>>> like
> > > > >> >>>>>>>>>> the first full build
after the merge [1] failed.  It
> > looked
> > > > >> >>> like
> > > > >> >>>>>>>>>> something hung along
the way, and that most of the
> > previous
> > > > >> >>>> builds
> > > > >> >>>>>>> had
> > > > >> >>>>>>>>>> failed for various reasons.
:(
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> I kicked it off again
have it do another try.  If it is
> > mob
> > > > >> >>>> related
> > > > >> >>>>>>>> we'll
> > > > >> >>>>>>>>>> take hunt it down and
take care of it.
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> Jon.
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> [1] https://builds.apache.org/job/HBase-TRUNK/6672/
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> On Wed, Jul 22, 2015
at 1:16 PM, Jonathan Hsieh <
> > > > >> >>>> jon@cloudera.com>
> > > > >> >>>>>>>>> wrote:
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>>> I've merged the
code in to master.  Thanks for all the
> > > hard
> > > > >> >>>> work
> > > > >> >>>>>>>>>>> Jingcheng and thanks
to all who have been involved
> with
> > > > >> >>>> reviews,
> > > > >> >>>>>>>>>>> discussion, and
voting!
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> Jon
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> On Wed, Jul 22,
2015 at 12:45 AM, Jingcheng Du <
> > > > >> >>>>>>>> jingcheng.du@intel.com>
> > > > >> >>>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>>> Hi all,
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>> The vote passes
with 8 +1s and no -1. Thanks all for
> > > > >> >>> guiding,
> > > > >> >>>>>>> helping
> > > > >> >>>>>>>>> and
> > > > >> >>>>>>>>>>>> voting!
> > > > >> >>>>>>>>>>>> We will work
on the merge activities and will let
> guys
> > > > >> >> know
> > > > >> >>>> about
> > > > >> >>>>>>> the
> > > > >> >>>>>>>>>>>> detailed plan
for merge time.
> > > > >> >>>>>>>>>>>> And thanks Jon
for helping merge this branch to
> trunk!
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>> Regards,
> > > > >> >>>>>>>>>>>> Jingcheng
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>> --
> > > > >> >>>>>>>>>>>> View this message
in context:
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/RESULT-VOTE-Merge-branch-hbase-11339-HBase-MOB-to-trunk-tp4073446.html
> > > > >> >>>>>>>>>>>> Sent from the
HBase Developer mailing list archive at
> > > > >> >>>> Nabble.com.
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> --
> > > > >> >>>>>>>>>>> // Jonathan Hsieh
(shay)
> > > > >> >>>>>>>>>>> // HBase Tech Lead,
Software Engineer, Cloudera
> > > > >> >>>>>>>>>>> // jon@cloudera.com
// @jmhsieh
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> --
> > > > >> >>>>>>>>>> // Jonathan Hsieh (shay)
> > > > >> >>>>>>>>>> // HBase Tech Lead,
Software Engineer, Cloudera
> > > > >> >>>>>>>>>> // jon@cloudera.com
// @jmhsieh
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>> --
> > > > >> >>>>>>>>> // Jonathan Hsieh (shay)
> > > > >> >>>>>>>>> // HBase Tech Lead, Software
Engineer, Cloudera
> > > > >> >>>>>>>>> // jon@cloudera.com // @jmhsieh
> > > > >> >>>>>>>>
> > > > >> >>>>>>>>
> > > > >> >>>>>>>>
> > > > >> >>>>>>>> --
> > > > >> >>>>>>>> Best regards,
> > > > >> >>>>>>>>
> > > > >> >>>>>>>>   - Andy
> > > > >> >>>>>>>>
> > > > >> >>>>>>>> Problems worthy of attack prove
their worth by hitting
> > back.
> > > -
> > > > >> >>> Piet
> > > > >> >>>>>> Hein
> > > > >> >>>>>>>> (via Tom White)
> > > > >> >>>>
> > > > >> >>>>
> > > > >> >>>>
> > > > >> >>>> --
> > > > >> >>>> Sean
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >> --
> > > > >> >> Best regards,
> > > > >> >>
> > > > >> >>   - Andy
> > > > >> >>
> > > > >> >> Problems worthy of attack prove their worth by hitting
back. -
> > Piet
> > > > Hein
> > > > >> >> (via Tom White)
> > > > >> >>
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > // Jonathan Hsieh (shay)
> > > > > // HBase Tech Lead, Software Engineer, Cloudera
> > > > > // jon@cloudera.com // @jmhsieh
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > >    - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>



-- 
// Jonathan Hsieh (shay)
// HBase Tech Lead, Software Engineer, Cloudera
// jon@cloudera.com // @jmhsieh

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message