hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nigel Daley <nige.da...@gmail.com>
Subject Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)
Date Wed, 24 Jun 2015 14:18:35 GMT
+1 for a separate project and going directly to TLP if possible (as Hadoop itself did when
split out of Nutch)

+1 for having language discussions once it's a TLP :-)

Cheers,
Nigel

> On Jun 22, 2015, at 1:55 PM, Andrew Purtell <apurtell@apache.org> wrote:
> 
>> On Mon, Jun 22, 2015 at 1:03 PM, Nick Dimiduk <ndimiduk@gmail.com> wrote:
>> 
>> On Mon, Jun 22, 2015 at 12:43 PM, Colin P. McCabe <cmccabe@apache.org>
>> wrote:
>> 
>>> You mentioned that "most of our project will be focused on shell
>>> scripts" I guess based on the existing test-patch code.  Allen did a
>>> lot of good work in this area recently.  I am curious if you evaluated
>>> languages such as Python or Node.js for this use-case.  Shell scripts
>>> can get a little... tricky beyond a certain size.  On the other hand,
>>> if we are standardizing on shell, which shell and which version?
>>> Perhaps bash 3.5+?
>> 
>> I'll also add that shell is not helpful for a cross-platform set of
>> tooling. I recently added a daemon to Apache Phoenix; an explicit
>> requirement was Windows support. I ended up implementing a solution in
>> python because that environment is platform-agnostic and still systems-y
>> enough. I think this is something this project should seriously consider.
> 
> In my opinion, historically, test-patch hasn't needed to be cross platform
> because the only first class development environment for Hadoop has been
> Linux. Growing beyond this could absolutely be one focus of Yetus should
> that be a consensus goal of the community. The seed of the project, though,
> is today's test-patch, which is implemented in bash. That's where we are
> today. Language "discussions" (smile) can and should be forward looking.
> 
> 
>> On Mon, Jun 22, 2015 at 1:03 PM, Nick Dimiduk <ndimiduk@gmail.com> wrote:
>> 
>> On Mon, Jun 22, 2015 at 12:43 PM, Colin P. McCabe <cmccabe@apache.org>
>> wrote:
>> 
>>> You mentioned that "most of our project will be focused on shell
>>> scripts" I guess based on the existing test-patch code.  Allen did a
>>> lot of good work in this area recently.  I am curious if you evaluated
>>> languages such as Python or Node.js for this use-case.  Shell scripts
>>> can get a little... tricky beyond a certain size.  On the other hand,
>>> if we are standardizing on shell, which shell and which version?
>>> Perhaps bash 3.5+?
>> 
>> I'll also add that shell is not helpful for a cross-platform set of
>> tooling. I recently added a daemon to Apache Phoenix; an explicit
>> requirement was Windows support. I ended up implementing a solution in
>> python because that environment is platform-agnostic and still systems-y
>> enough. I think this is something this project should seriously consider.
>> 
>> -n
>> 
>> On Tue, Jun 16, 2015 at 7:55 PM, Sean Busbey <busbey@cloudera.com> wrote:
>>>> I'm going to try responding to several things at once here, so
>> apologies
>>> if
>>>> I miss anyone and sorry for the long email. :)
>>>> 
>>>> 
>>>> On Tue, Jun 16, 2015 at 3:44 PM, Steve Loughran <
>> stevel@hortonworks.com>
>>>> wrote:
>>>> 
>>>>> I think it's good to have a general build/test process projects can
>>> share,
>>>>> so +1 to pulling it out. You should get help from others.
>>>>> 
>>>>> regarding incubation, it is a lot of work, especially for something
>>> that's
>>>>> more of an in-house tool than an artifact to release and redistribute.
>>>>> 
>>>>> You can't just use apache labs or the build project's repo to work on
>>> this?
>>>>> 
>>>>> if you do want to incubate, we may want to nominate the hadoop project
>>> as
>>>>> the monitoring PMC, rather than incubator@.
>>>>> 
>>>>> -steve
>>>> Important note: we're proposing a board resolution that would directly
>>> pull
>>>> this code base out into a new TLP; there'd be no incubator, we'd just
>>>> continue building community and start making releases.
>>>> 
>>>> The proposed PMC believes the tooling we're talking about has direct
>>>> applicability to projects well outside of the ASF. Lot's of other open
>>>> source projects run on community contributions and have a general need
>>> for
>>>> better QA tools. Given that problem set and the presence of a community
>>>> working to solve it, there's no reason this needs to be treated as an
>>>> in-house build project. We certainly want to be useful to ASF projects
>>> and
>>>> getting them on-board given our current optimization for ASF infra will
>>>> certainly be easier, but we're not limited to that (and our current
>>>> prerequisites, a CI tool and jira or github, are pretty broadly
>>> available).
>>>> 
>>>> 
>>>>> On Tue, Jun 16, 2015 at 10:13 AM, Nick Dimiduk <ndimiduk@apache.org>
>>>> wrote:
>>>> 
>>>>> 
>>>>> Since we're tossing out names, how about Apache Bootstrap? It's a
>>>>> meta-project to help other projects get off the ground, after all.
>>>> 
>>>> 
>>>> There's already a web development framework named Bootstrap[1]. It's
>> also
>>>> used by several ASF projects, so I think it best to avoid the
>> confusion.
>>>> 
>>>> The name is, of course, up to the proposed PMC. As a bit of background,
>>> the
>>>> current name Yetus fulfills Allen's desire to have something shell
>>> related
>>>> and my desire to have a project that starts with Y (there are currently
>>> no
>>>> ASF projects that start with Y). The universe of names that fill in
>> these
>>>> two is very small, AFAICT. I did a brief suitability search and didn't
>>> find
>>>> any blockers.
>>>> 
>>>> 
>>>> On Tue, Jun 16, 2015 at 11:59 AM, Allen Wittenauer <aw@altiscale.com>
>>>> wrote:
>>>> 
>>>>> 
>>>>> Since a couple of people have brought it up:
>>>>> 
>>>>>        I think the release question is probably one of the big
>> question
>>>>> marks.  Other than tar balls, how does something like this actually
>> get
>>>>> used downstream?
>>>>> 
>>>>>        For test-patch, in particular, I have a few thoughts on this:
>>>>> 
>>>>> Short term:
>>>>> 
>>>>>        * Projects that want to move RIGHT NOW would modify their
>>> Jenkins
>>>>> jobs to checkout from the Yetus repo (preferably at a well known tag
>> or
>>>>> branch) in one directory and their project repo in another directory.
>>> Then
>>>>> it’s just a matter of passing the correct flags to test-patch.  This
>> is
>>>>> pretty much how I’ve been personally running test-patch for about 6
>>> months
>>>>> now. Under Jenkins, we’ve seen this work with NiFi (incubating)
>> already.
>>>>> 
>>>>>        * Create a stub version of test-patch that projects could
>> check
>>>>> into their repo, replacing the existing test-patch.  This stub version
>>>>> would git clone from either ASF or github and then execute test-patch
>>>>> accordingly on demand.  With the correct smarts, it could make sure it
>>> has
>>>>> a cached version to prevent continual clones.
>>>>> 
>>>>> Longer term:
>>>>> 
>>>>>        * I’ve been toying with the idea of (ab)using Java repos and
>>>>> packaging as a transportation layer, either in addition or in
>>> combination
>>>>> with something like a maven plugin.  Something like this would clearly
>>> be
>>>>> better for offline usage and/or to lower the network traffic.
>>>> 
>>>> It's important that the project follow ASF guidelines on publishing
>>>> releases[2]. So long as we publish releases to the distribution
>>> directory I
>>>> think we'd be fine having folks work off of the corresponding tag. I'm
>>> not
>>>> sure there's much reason to do that, however. A Jenkins job can just as
>>>> easily grab a release tarball as a git tag and we're not talking about
>> a
>>>> large amount of stuff. The kind of build setup that Chris N mentioned
>> is
>>>> also totally doable now that there's a build description DSL for
>>> Jenkins[3].
>>>> 
>>>> For individual developers, I don't see any reason we can't package
>> things
>>>> up as a tool, similar to how findbugs or shellcheck work. We can make
>> OS
>>>> packages (or homebrew for OS X) if we want to make stand alone
>>> installation
>>>> on developer machines real easy. Those same packages could be installed
>>> on
>>>> the ASF build machines, provided some ASF project wanted to make use of
>>>> Yetus.
>>>> 
>>>> Having releases will incur some turn around time for when folks want to
>>> see
>>>> fixes, but that's a trade off around release cadence we can work out
>>> longer
>>>> term.
>>>> 
>>>> I would like to have one or two projects that can work off of the
>>> bleeding
>>>> edge repo, but we'd have to get that to mesh with foundation policy. My
>>> gut
>>>> tells me we should be able to come up with an agreement that makes
>> such a
>>>> project "part of the development community" but the specifics will have
>>> to
>>>> be worked out.
>>>> 
>>>> 
>>>>> On Tue, Jun 16, 2015 at 4:41 AM, Jonathan Hsieh <jon@cloudera.com>
>>>> wrote:
>>>> 
>>>>> How would we have to add testing to our pre commit testing?
>>>>> 
>>>>> Each project has likely customized their own core commit scripts
>>> (multiple
>>>>> Jvms versions, checkstyle, javadoc exceptions etc.). We should
>> probably
>>>>> soliciit interest from other projects who already have fancy precommit
>>>>> tests beyond HBase/Hadoop too.
>>>> 
>>>> I'm not sure if Allen's response above answered this for you. The
>> current
>>>> state of test-patch has a plugin system for adding new tests. It is
>> also
>>>> customizable to handle project idiosyncrasies (atleast the ones we've
>>> seen
>>>> so far, like unspecified module dependencies, multiple projects per
>> repo,
>>>> or use of ant). Releases will naturally include docs on how to leverage
>>>> those to customize what a particular project needs tested pre-commit.
>>>> 
>>>> Given the nature of the work Yetus is hoping to enable, I think it's
>> safe
>>>> to assume the project community is going to be doing a fair bit of
>>> outreach
>>>> to help show projects outside of HBase/Hadoop how things can work for
>>> them.
>>>> 
>>>> 
>>>> 
>>>> [1]: http://getbootstrap.com/
>>>> [2]: http://www.apache.org/dev/release.html
>>>> [3]: https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin
>>>> 
>>>> --
>>>> Sean
> 
> 
> 
> -- 
> Best regards,
> 
>   - Andy
> 
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)

Mime
View raw message