hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Rawson <ryano...@gmail.com>
Subject Re: Discussion: Move contribs out of hbase?
Date Sun, 31 Jan 2010 01:03:56 GMT
What does it take to move a contrib into core?

I would think that a committer must be able to fix any problem with it
at minimum...

On Sat, Jan 30, 2010 at 4:57 PM, Stack <stack@duboce.net> wrote:
> On Thu, Jan 28, 2010 at 3:08 PM, Joydeep Sarma <jsensarma@gmail.com> wrote:
>> fwiw - i agree with Dhruba's take. i can't imagine (for example)
>> hadoop shipping without hadoop-streaming (which is contrib). and it's
>> awesome that hadoop changes that break streaming are caught early and
>> often.
>
> I suggest that contribs we deem critical should be pulled up out of
> contrib and moved up to core.
>
> Regards changes in core breaking contrib early, contribs moved out to
> other repositories can retain this 'benefit' by building against
> snapshots of the core TRUNK via built-in ivy/maven facility, only now,
> the onus for updating the moved contrib is no longer inline with core
> dev and instead has been shifted  to the moved contrib maintainer.  It
> is up to him or her to update to match core changes at their pleasure
> or to report on core changes that cause fundamental distress in
> contrib.
>
>
>> it makes sense this analog doesn't apply to all contrib projects -
>> perhaps they shouldn't be in contrib to begin with?
>
> Yes.  There is some of this going on.  IMO, all currently have
> sufficient character living apart outside contrib.
>
>
> alternatively - to
>> extend Dhruba's idea - one could break up contrib projects into those
>> that are critical (like streaming would be to hadoop and i imagine the
>> replication stuff would be to hbase) - and those that are non-critical
>> (and not build these by default).
>>
> We could do this but to my mind, I can't help but think that we are
> just piling the work integrating contribs to the period that just
> precedes a release.
>
>
>> the only reason to have non-critical contrib projects in the codebase
>> seems to be to give young/new projects exposure. i am not sure whether
>> this is a good enough reason - perhaps those with some historical
>> perspective have better idea on this.
>>
>
> You have it right.  I think it fair to say that that explains how we
> ended up with our current set of contribs.  At least one contrib has
> been in hbase more than a year now.  Its time we held a graduation for
> it at least?
>
> St.Ack
>
>>
>> On Thu, Jan 28, 2010 at 2:35 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>>> Part of the problem with contrib is fixing the deeper problems during
>>> major version changes (eg: new file format in 0.20) requires
>>> specialized knowledge that only the contrib author has. Then they must
>>> write a patch, submit it to a hbase committer, etc.  If the contrib
>>> was in a non-ASF repo, the author could fix it much faster.
>>>
>>> That's my theory at least?
>>>
>>> On Thu, Jan 28, 2010 at 2:27 PM, Dhruba Borthakur <dhruba@gmail.com> wrote:
>>>> Hi stack,
>>>>
>>>> I have seen this issue crop up in the HDFS  repository too.
>>>>
>>>> The issue that you have brought up is that code changes to core sometimes
>>>> needs lots of changes to contrib to make it compile. This is a big problem,
>>>> especially when all developers are not intricately familiar with each and
>>>> every contrib module. However, the big plus is that when a HBase release
is
>>>> made, it has all the contrib modules in it, and it makes it easier for a
>>>> user to start using any of those contrib modules right away. Can we get the
>>>> best of both worlds if we do something like this:
>>>>
>>>>  Contribs are not first-class citizens. The build script can be setup to
>>>> ignore build failures in contrib modules. The hope is that when a release
is
>>>> about to be cut, contrib owners will submit fixes to any build problems if
>>>> they want their contrib to be shipped as part of the Hbase release. will
>>>> this solve your problem?
>>>>
>>>> thanks,
>>>> dhruba
>>>>
>>>>
>>>>
>>>> On Thu, Jan 28, 2010 at 12:42 PM, Ryan Rawson <ryanobjc@gmail.com>
wrote:
>>>>
>>>>> +1
>>>>>
>>>>> I think when we are ready, we should publish the replication as a
>>>>> contrib up on github. I really like github, its way better than any
>>>>> other system I've seen (in part due to git of course).
>>>>>
>>>>> I do like the notion of officially released contribs that are included
>>>>> at specific version numbers with the hbase release. Hbase core 0.21.0
>>>>> with TH, ITH at version 0.1 or whatever.
>>>>>
>>>>> -ryan
>>>>>
>>>>> On Thu, Jan 28, 2010 at 12:38 PM, Stack <stack@duboce.net> wrote:
>>>>> > I'd like to start a discussion on moving src/contrib out of hbase.
>>>>> > Keep reading if you have an opinion.
>>>>> >
>>>>> > I'd like to suggest that we undo the notion of hbase contribs for
the
>>>>> > following reasons:
>>>>> >
>>>>> > + In my experience, they present a friction on changes to core as
any
>>>>> > significant core change tends to ripple down into the contribs whether
>>>>> > its code or infrastructure/build changes.  Usually what happens
then
>>>>> > is a non-expert in the contrib code is making edits -- often radical
>>>>> > -- to code they are not completely up on and are a little frustrated
>>>>> > that they have to do it.  Bad.
>>>>> > + A few of our contribs are maintained by non-committers.  This
means
>>>>> > it takes a committers time getting in updates.  The owner is at
mercy
>>>>> > of the committer making wanted changes.  The committer is consumed
>>>>> > reviewing and making update.  This indirection hurts at both ends
(We
>>>>> > could discuss making contrib owners committers on their contrib
only
>>>>> > but that'd be a bit of bureaucratic nightmare and a burden on the
>>>>> > hadoop pmc to vote on granting access to a subprojects, contrib.
 Its
>>>>> > tough enough getting hadoop pmc to vote on hbase committers.  There
is
>>>>> > no precedent in other project, to my knowledge).
>>>>> > + Contribs and core evolve at different rates.  They should not
be
>>>>> > constrained by core release schedule (or the opposite, core should
not
>>>>> > be held up because fixes in contrib are wanting).
>>>>> >
>>>>> > I suggest that current contribs be moved out of hbase up to standalone
>>>>> > github or google code projects (witness how hbql does it).  Previous
>>>>> > to our move to Ivy (and possibly soon, Maven), asking contribs be
>>>>> > standalone was a pain as they'd have to check in hbase jars and
all of
>>>>> > dependencies and then move these forward over time.  Now that we
are
>>>>> > Ivy-ized, contribs just need write a bit of ivy.xml and it'll take
>>>>> > care of pulling dependencies.
>>>>> >
>>>>> > I'd imagine support for contribs would go on as it does now with
>>>>> > queries up on hbase mailing list and help out on IRC.  We'd give
>>>>> > contribs first-class billing up on home page.  Popular contribs
might
>>>>> > run their own mailing lists, etc...
>>>>> >
>>>>> > We should chat but some contribs should be pulled up into core.
>>>>> > Thrift was core.  Talk was to move current thrift update out to
>>>>> > contrib.  I think now it should just stay in core.  Stargate perhaps
>>>>> > should come up into core?
>>>>> >
>>>>> > What do folks think?
>>>>> > St.Ack
>>>>> >
>>>>> > P.S. I've fostered the contrib notion in the past.  I've since
had a
>>>>> > change of heart.  Please pardon my flip.
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Connect to me at http://www.facebook.com/dhruba
>>>>
>>>
>>
>

Mime
View raw message