Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Content-Type: text/plain;
	charset=utf-8
Mime-Version: 1.0 (1.0)
Subject: Re: MOB in branch-1? (Re: [RESULT][VOTE] Merge branch hbase-11339
 HBase MOB to trunk)
From: Ted Yu <yuzhihong@gmail.com>
In-Reply-To: <97556505-2173-4849-8CCB-48488F167F61@gmail.com>
Date: Mon, 29 Feb 2016 20:20:15 -0800
Cc: "dev@hbase.apache.org" <dev@hbase.apache.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <BB7DDD8F-FFC6-4F70-94F0-BD30C891EA89@gmail.com>
References: 
 <CAPQQKxfnmY4h2UJKFrrzmWFaL5t8qnTzPAfoUdFctCay56Nrxg@mail.gmail.com>
 <CADcMMgFsEoi9KdPsBUq7Fp2b8wDTaFA2u1ZRtfGXT50qjSGh8A@mail.gmail.com>
 <CALvzVW1+7Kuj3LqPrcLCFmzN1iqqHeL-r5VN4H68+_=n=TbP4Q@mail.gmail.com>
 <CAGHyZ6KMd6QHW1-Pe3MpbL2DQG+0iKvHi503-vOFE8bfpe4zyg@mail.gmail.com>
 <CAPQQKxckWTMKaVQ2ZUk_PUJtszf0Vd__dXZY36khk0i04-UwNg@mail.gmail.com>
 <CA+RK=_B-HFSScfGtet8fTn__cJubxbRPYRifpqB2FJbnH=26NQ@mail.gmail.com>
 <CALte62xMKZx7vWs=Ph3Y3+X4doa-Cduo1FqFqqvcBpEG9-93WQ@mail.gmail.com>
 <EF3CF06F-9D1A-4AD5-A3B1-0BA5EE62D556@gmail.com>
 <CAAha9a2dt55Qqt9bups0oLPau0DjDSPXo3Csaoot8oPtLeNwsw@mail.gmail.com>
 <CAN5cbe7hxE7MFPWFe7S6N1OgUP7wfVWPSRVyyE5OhwfbsX7SwA@mail.gmail.com>
 <CA+RK=_BEkqtkxjrkEOtwMQcOMOzqKqb_4FTPK09wqDXnv+VtaQ@mail.gmail.com>
 <CALte62zDqHGyJYTw_nYNv66J730OK-tAmBtrLZ14giprK9_HZw@mail.gmail.com>
 <CA+RK=_Bas2130ZAH=E7mQj5WRbbE+ST-j7=P40WuONTJe7T_kQ@mail.gmail.com>
 <CAAha9a39bzEUXArBQow-yAc9d+CXxfVzpf5jLQ6nDCCr0suF6Q@mail.gmail.com>
 <97556505-2173-4849-8CCB-48488F167F61@gmail.com>
To: user@hbase.apache.org

If there is no objection, I will create a backport JIRA tomorrow and attach p=
atch.=20

Thanks

> On Feb 29, 2016, at 8:14 PM, Andrew Purtell <andrew.purtell@gmail.com> wro=
te:
>=20
> Thanks Jon. This is very helpful information for those of us who don't hav=
e visibility to these users. Answers my question.=20
>=20
>=20
>> On Feb 29, 2016, at 6:56 PM, Jonathan Hsieh <jon@cloudera.com> wrote:
>>=20
>> The feature is definitely not abandoned -- we have few sizable customers a=
t
>> PB scale that I can recall off the top of my head that have been using it=

>> for over 8-12 months in the version backported to CDH (we backported an
>> early version back in oct/14 (CDH5.2), and updated with the recent more
>> recent changes in roughly may/15 (CDH5.4).   These are used primarily to
>> store documents -- think PDF files. They are pretty happy -- the customer=

>> reported that it was slightly slower on the write side (10-15%) than a
>> competing system but significantly faster on the read side (3x-4x
>> throughput).
>>=20
>> Over the holidays we shook out a semi-rare data loss issue with mob
>> compaction that had this as the root cause[1] -- (since pointers are only=

>> stored in the "normal hfiles" the volume of datas on this case was fairly=

>> large).
>>=20
>> We've been working on trying to get at least one of them to present at
>> hbasecon, but I'm not reviewing submissions this year and don't know if
>> they made it or not.
>>=20
>> Jon.
>>=20
>> [1] https://issues.apache.org/jira/browse/HBASE-15035
>>=20
>> On Fri, Feb 26, 2016 at 12:27 PM, Andrew Purtell <apurtell@apache.org>
>> wrote:
>>=20
>>> =E2=80=8BI think we need at least one success story or one very interest=
ed user
>>> with a real project on the line to justify a backport. Otherwise it's a
>>> feature without any users - technically, abandoned. =E2=80=8B
>>>=20
>>>> On Fri, Feb 26, 2016 at 10:11 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>>>>=20
>>>> I am interested in hearing about user experience with MOB feature as
>>> well.
>>>>=20
>>>> In my opinion, this feature is a nice addition to branch-1.
>>>>=20
>>>> On Wed, Feb 10, 2016 at 1:45 PM, Andrew Purtell <apurtell@apache.org>
>>>> wrote:
>>>>=20
>>>>> +user@
>>>>>=20
>>>>> Is there anyone using the MOB feature in trunk for anything who can
>>>> comment
>>>>> on how well it's been working out? Intel folks maybe?
>>>>>=20
>>>>> On Thu, Jan 21, 2016 at 7:46 AM, Sean Busbey <busbey@apache.org>
>>> wrote:
>>>>>=20
>>>>>> The last time MOB on branch-1 came up, folks were concerned that it
>>>>>> wasn't stable enough in master yet. Is that still the case?
>>>>>>=20
>>>>>> Can we get a [DISCUSS] flagged thread to see what, if anything, folks=

>>>>>> would like to see gate inclusion in branch-1?
>>>>>>=20
>>>>>> On Tue, Jan 19, 2016 at 7:31 PM, Jonathan Hsieh <jon@cloudera.com>
>>>>> wrote:
>>>>>>> +1 to 1.2 being feature complete corrently.  There has already
>>> been a
>>>>>>> release candidate and folks are burning down the blockers currently
>>>> to
>>>>>> prep
>>>>>>> for the next RC.
>>>>>>>=20
>>>>>>> I like the idea of mob and sparkonhbase for 1.3.  I'm more
>>>> comfortable
>>>>>> with
>>>>>>> sparkonhbase -- it is a new module and thus not as invasive.
>>>>>>>=20
>>>>>>> Jon.
>>>>>>>=20
>>>>>>> On Tue, Jan 19, 2016 at 4:55 PM, Andrew Purtell <
>>>>>> andrew.purtell@gmail.com>
>>>>>>> wrote:
>>>>>>>=20
>>>>>>>> Pretty sure Sean expressed 1.2 is feature complete and I'd support
>>>>> that.
>>>>>>>> Can we wait for 1.3 for MOB ? Can look at Spark connector then
>>> too.
>>>>>>>>=20
>>>>>>>>> On Jan 19, 2016, at 4:52 PM, Ted Yu <yuzhihong@gmail.com>
>>> wrote:
>>>>>>>>>=20
>>>>>>>>> Looks like 1.2.0 RC is in near future.
>>>>>>>>>=20
>>>>>>>>> I wonder if it is time to revive this thread (due to customer
>>>>>> interest).
>>>>>>>>>=20
>>>>>>>>> As far as I can tell, the Mob related tests have been passing in
>>>> the
>>>>>>>> recent
>>>>>>>>> past.
>>>>>>>>>=20
>>>>>>>>> Thanks
>>>>>>>>>=20
>>>>>>>>>> On Wed, Oct 28, 2015 at 2:07 PM, Andrew Purtell <
>>>>> apurtell@apache.org
>>>>>>>=20
>>>>>>>> wrote:
>>>>>>>>>>=20
>>>>>>>>>> I haven't heard an user answer in the affirmative to wanting
>>> it.
>>>>>>>>>>=20
>>>>>>>>>> I'll volunteer to RM 1.3, whenever we need it. Premature to
>>> have
>>>>> that
>>>>>>>>>> discussion without 1.2 even out the door yet, though.
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>> On Wed, Oct 28, 2015 at 1:01 PM, Stephen Jiang <
>>>>>> syuanjiangdev@gmail.com
>>>>>>>>>=20
>>>>>>>>>> wrote:
>>>>>>>>>>=20
>>>>>>>>>>> Actually, it is actively changing in master branch on MOB
>>>> feature
>>>>>> made
>>>>>>>> me
>>>>>>>>>>> think about: if we ever want to port MOB feature to branch-1,
>>>> now
>>>>>> is a
>>>>>>>>>> good
>>>>>>>>>>> time.  We can commit changes in both branches; otherwise, we
>>>>>> probably
>>>>>>>>>> would
>>>>>>>>>>> miss some commits when we port MOB to branch-1 in a late time.
>>>>>>>>>>>=20
>>>>>>>>>>> I am more thinking about 1.3 release (certainly not 1.2),
>>> which
>>>> we
>>>>>>>> still
>>>>>>>>>>> have some time to stabilize and allow interesting party to
>>> play
>>>>> with
>>>>>>>> the
>>>>>>>>>>> feature and give feedback.
>>>>>>>>>>>=20
>>>>>>>>>>> Thanks
>>>>>>>>>>> Stephen
>>>>>>>>>>>=20
>>>>>>>>>>> PS. given the features we discussed in 2.0.0 in the last
>>>> community
>>>>>>>>>> meeting,
>>>>>>>>>>> I think it would not release earlier than 1.3 :-), unless we
>>>>>>>>>> intentionally
>>>>>>>>>>> not find a release manager for 1.3.
>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>>> On Wed, Oct 28, 2015 at 9:09 AM, Sean Busbey <
>>>>> busbey@cloudera.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>=20
>>>>>>>>>>>> It's practically November. Matteo, are you up for a thread on
>>>>>> target
>>>>>>>>>>>> dates for 2.0.0 to start RCs?
>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>>>>>> On Wed, Oct 28, 2015 at 11:02 AM, Elliott Clark <
>>>>> eclark@apache.org
>>>>>>>=20
>>>>>>>>>>> wrote:
>>>>>>>>>>>>> I feel the same lets keep branch-1 stable, and work towards
>>> a
>>>>>> faster
>>>>>>>>>>>> 2.0.0.
>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> On Tue, Oct 27, 2015 at 4:52 PM, Stack <stack@duboce.net>
>>>>> wrote:
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> IMO, MOB is still not settled in Master. It has a bunch of
>>>>> flakey
>>>>>>>>>>> tests
>>>>>>>>>>>>>> that are getting fixed by Jingcheng or I've disabled them
>>>> till
>>>>>>>>>> someone
>>>>>>>>>>>> has
>>>>>>>>>>>>>> time to look at them. There is also a load of duplicated
>>> code
>>>>>> that
>>>>>>>>>> is
>>>>>>>>>>>> being
>>>>>>>>>>>>>> cleaned up (Matteo).
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> Its not ready to go back to branch-1 IMO.
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> Are there users who'd like it backported?
>>>>>>>>>>>>>> St.Ack
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> On Mon, Oct 26, 2015 at 10:55 AM, Stephen Jiang <
>>>>>>>>>>>> syuanjiangdev@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> Hello, guys, the MOB is in master branch.  I saw bug fixes
>>>>>>>>>> happening
>>>>>>>>>>>> in
>>>>>>>>>>>>>>> master branch.
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> I just wonder whether there is a plan to put MOB in
>>>>> branch-1.  I
>>>>>>>>>> am
>>>>>>>>>>>>>> afraid
>>>>>>>>>>>>>>> if we don't do it now, it would be harder in the future to
>>>>> back
>>>>>>>>>> port
>>>>>>>>>>>> if
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>> decide to do it in a late time.
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>> Stephen
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> On Thu, Jul 23, 2015 at 1:15 PM, Andrew Purtell <
>>>>>>>>>>> apurtell@apache.org>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>> Thanks Jon.
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>> When I'm back in the office I'll check out master and
>>> have
>>>> a
>>>>>>>>>> look
>>>>>>>>>>>> into
>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>> locally repeatable test failures. Anyway in my opinion at
>>>>> this
>>>>>>>>>>>> point it
>>>>>>>>>>>>>>>> would make the most sense for us to keep the MOB changes
>>> in
>>>>> on
>>>>>>>>>>>> master
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> deal with any fallout in follow on issues. I think all
>>> who
>>>>>> voted
>>>>>>>>>>> +1
>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> this change were aware that large changes like this can
>>>> have
>>>>> a
>>>>>>>>>>>>>>> temporarily
>>>>>>>>>>>>>>>> destabilizing effect. As long as the MOB devs are around
>>> to
>>>>>> help
>>>>>>>>>>>> clean
>>>>>>>>>>>>>>> up,
>>>>>>>>>>>>>>>> we should be good!
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>> On Wed, Jul 22, 2015 at 8:09 PM, Jonathan Hsieh <
>>>>>>>>>> jon@cloudera.com
>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> I had two clean full builds/unit test on my internal
>>> setup
>>>>> and
>>>>>>>>>>> the
>>>>>>>>>>>>>>> latest
>>>>>>>>>>>>>>>>> build went back to ~4325 total tests and failures on
>>>>> Procedure
>>>>>>>>>>>> relate
>>>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>>> cases.
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> I don't think mob is responsible for these failures.
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> Jon.
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> On Wed, Jul 22, 2015 at 4:32 PM, Jonathan Hsieh <
>>>>>>>>>>> jon@cloudera.com
>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> Although the the precommit buiid passed, and the
>>>>> compilation
>>>>>>>>>>> and
>>>>>>>>>>>>>> mob
>>>>>>>>>>>>>>>>>> testing I ran after before the merge was commited
>>> passed,
>>>>> It
>>>>>>>>>>>> looks
>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>> the first full build after the merge [1] failed.  It
>>>> looked
>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>> something hung along the way, and that most of the
>>>> previous
>>>>>>>>>>>> builds
>>>>>>>>>>>>>>> had
>>>>>>>>>>>>>>>>>> failed for various reasons. :(
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> I kicked it off again have it do another try.  If it is
>>>> mob
>>>>>>>>>>>> related
>>>>>>>>>>>>>>>> we'll
>>>>>>>>>>>>>>>>>> take hunt it down and take care of it.
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> Jon.
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> [1] https://builds.apache.org/job/HBase-TRUNK/6672/
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> On Wed, Jul 22, 2015 at 1:16 PM, Jonathan Hsieh <
>>>>>>>>>>>> jon@cloudera.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>> I've merged the code in to master.  Thanks for all the
>>>>> hard
>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>>>> Jingcheng and thanks to all who have been involved
>>> with
>>>>>>>>>>>> reviews,
>>>>>>>>>>>>>>>>>>> discussion, and voting!
>>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>> On Wed, Jul 22, 2015 at 12:45 AM, Jingcheng Du <
>>>>>>>>>>>>>>>> jingcheng.du@intel.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>>> The vote passes with 8 +1s and no -1. Thanks all for
>>>>>>>>>>> guiding,
>>>>>>>>>>>>>>> helping
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> voting!
>>>>>>>>>>>>>>>>>>>> We will work on the merge activities and will let
>>> guys
>>>>>>>>>> know
>>>>>>>>>>>> about
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> detailed plan for merge time.
>>>>>>>>>>>>>>>>>>>> And thanks Jon for helping merge this branch to
>>> trunk!
>>>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>> Jingcheng
>>>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> View this message in context:
>>> http://apache-hbase.679495.n3.nabble.com/RESULT-VOTE-Merge-branch-hbase-=
11339-HBase-MOB-to-trunk-tp4073446.html
>>>>>>>>>>>>>>>>>>>> Sent from the HBase Developer mailing list archive at
>>>>>>>>>>>> Nabble.com.
>>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> // Jonathan Hsieh (shay)
>>>>>>>>>>>>>>>>>>> // HBase Tech Lead, Software Engineer, Cloudera
>>>>>>>>>>>>>>>>>>> // jon@cloudera.com // @jmhsieh
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> // Jonathan Hsieh (shay)
>>>>>>>>>>>>>>>>>> // HBase Tech Lead, Software Engineer, Cloudera
>>>>>>>>>>>>>>>>>> // jon@cloudera.com // @jmhsieh
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> // Jonathan Hsieh (shay)
>>>>>>>>>>>>>>>>> // HBase Tech Lead, Software Engineer, Cloudera
>>>>>>>>>>>>>>>>> // jon@cloudera.com // @jmhsieh
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>> - Andy
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>> Problems worthy of attack prove their worth by hitting
>>>> back.
>>>>> -
>>>>>>>>>>> Piet
>>>>>>>>>>>>>> Hein
>>>>>>>>>>>>>>>> (via Tom White)
>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>>>>>> --
>>>>>>>>>>>> Sean
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>> --
>>>>>>>>>> Best regards,
>>>>>>>>>>=20
>>>>>>>>>> - Andy
>>>>>>>>>>=20
>>>>>>>>>> Problems worthy of attack prove their worth by hitting back. -
>>>> Piet
>>>>>> Hein
>>>>>>>>>> (via Tom White)
>>>>>>>=20
>>>>>>>=20
>>>>>>>=20
>>>>>>> --
>>>>>>> // Jonathan Hsieh (shay)
>>>>>>> // HBase Tech Lead, Software Engineer, Cloudera
>>>>>>> // jon@cloudera.com // @jmhsieh
>>>>>=20
>>>>>=20
>>>>>=20
>>>>> --
>>>>> Best regards,
>>>>>=20
>>>>>  - Andy
>>>>>=20
>>>>> Problems worthy of attack prove their worth by hitting back. - Piet
>>> Hein
>>>>> (via Tom White)
>>>=20
>>>=20
>>>=20
>>> --
>>> Best regards,
>>>=20
>>>  - Andy
>>>=20
>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein=

>>> (via Tom White)
>>=20
>>=20
>>=20
>> --=20
>> // Jonathan Hsieh (shay)
>> // HBase Tech Lead, Software Engineer, Cloudera
>> // jon@cloudera.com // @jmhsieh