hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <bus...@cloudera.com>
Subject Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)
Date Wed, 17 Jun 2015 02:55:17 GMT
I'm going to try responding to several things at once here, so apologies if
I miss anyone and sorry for the long email. :)


On Tue, Jun 16, 2015 at 3:44 PM, Steve Loughran <stevel@hortonworks.com>
wrote:

> I think it's good to have a general build/test process projects can share,
> so +1 to pulling it out. You should get help from others.
>
> regarding incubation, it is a lot of work, especially for something that's
> more of an in-house tool than an artifact to release and redistribute.
>
> You can't just use apache labs or the build project's repo to work on this?
>
> if you do want to incubate, we may want to nominate the hadoop project as
> the monitoring PMC, rather than incubator@.
>
> -steve
>
>
Important note: we're proposing a board resolution that would directly pull
this code base out into a new TLP; there'd be no incubator, we'd just
continue building community and start making releases.

The proposed PMC believes the tooling we're talking about has direct
applicability to projects well outside of the ASF. Lot's of other open
source projects run on community contributions and have a general need for
better QA tools. Given that problem set and the presence of a community
working to solve it, there's no reason this needs to be treated as an
in-house build project. We certainly want to be useful to ASF projects and
getting them on-board given our current optimization for ASF infra will
certainly be easier, but we're not limited to that (and our current
prerequisites, a CI tool and jira or github, are pretty broadly available).


On Tue, Jun 16, 2015 at 10:13 AM, Nick Dimiduk <ndimiduk@apache.org> wrote:

>
> Since we're tossing out names, how about Apache Bootstrap? It's a
> meta-project to help other projects get off the ground, after all.
>


There's already a web development framework named Bootstrap[1]. It's also
used by several ASF projects, so I think it best to avoid the confusion.

The name is, of course, up to the proposed PMC. As a bit of background, the
current name Yetus fulfills Allen's desire to have something shell related
and my desire to have a project that starts with Y (there are currently no
ASF projects that start with Y). The universe of names that fill in these
two is very small, AFAICT. I did a brief suitability search and didn't find
any blockers.


 On Tue, Jun 16, 2015 at 11:59 AM, Allen Wittenauer <aw@altiscale.com>
 wrote:

>
> Since a couple of people have brought it up:
>
>         I think the release question is probably one of the big question
> marks.  Other than tar balls, how does something like this actually get
> used downstream?
>
>         For test-patch, in particular, I have a few thoughts on this:
>
> Short term:
>
>         * Projects that want to move RIGHT NOW would modify their Jenkins
> jobs to checkout from the Yetus repo (preferably at a well known tag or
> branch) in one directory and their project repo in another directory.  Then
> it’s just a matter of passing the correct flags to test-patch.  This is
> pretty much how I’ve been personally running test-patch for about 6 months
> now. Under Jenkins, we’ve seen this work with NiFi (incubating) already.
>
>         * Create a stub version of test-patch that projects could check
> into their repo, replacing the existing test-patch.  This stub version
> would git clone from either ASF or github and then execute test-patch
> accordingly on demand.  With the correct smarts, it could make sure it has
> a cached version to prevent continual clones.
>
> Longer term:
>
>         * I’ve been toying with the idea of (ab)using Java repos and
> packaging as a transportation layer, either in addition or in combination
> with something like a maven plugin.  Something like this would clearly be
> better for offline usage and/or to lower the network traffic.
>

It's important that the project follow ASF guidelines on publishing
releases[2]. So long as we publish releases to the distribution directory I
think we'd be fine having folks work off of the corresponding tag. I'm not
sure there's much reason to do that, however. A Jenkins job can just as
easily grab a release tarball as a git tag and we're not talking about a
large amount of stuff. The kind of build setup that Chris N mentioned is
also totally doable now that there's a build description DSL for Jenkins[3].

For individual developers, I don't see any reason we can't package things
up as a tool, similar to how findbugs or shellcheck work. We can make OS
packages (or homebrew for OS X) if we want to make stand alone installation
on developer machines real easy. Those same packages could be installed on
the ASF build machines, provided some ASF project wanted to make use of
Yetus.

Having releases will incur some turn around time for when folks want to see
fixes, but that's a trade off around release cadence we can work out longer
term.

I would like to have one or two projects that can work off of the bleeding
edge repo, but we'd have to get that to mesh with foundation policy. My gut
tells me we should be able to come up with an agreement that makes such a
project "part of the development community" but the specifics will have to
be worked out.


On Tue, Jun 16, 2015 at 4:41 AM, Jonathan Hsieh <jon@cloudera.com> wrote:

> How would we have to add testing to our pre commit testing?
>
> Each project has likely customized their own core commit scripts (multiple
> Jvms versions, checkstyle, javadoc exceptions etc.). We should probably
> soliciit interest from other projects who already have fancy precommit
> tests beyond HBase/Hadoop too.
>

I'm not sure if Allen's response above answered this for you. The current
state of test-patch has a plugin system for adding new tests. It is also
customizable to handle project idiosyncrasies (atleast the ones we've seen
so far, like unspecified module dependencies, multiple projects per repo,
or use of ant). Releases will naturally include docs on how to leverage
those to customize what a particular project needs tested pre-commit.

Given the nature of the work Yetus is hoping to enable, I think it's safe
to assume the project community is going to be doing a fair bit of outreach
to help show projects outside of HBase/Hadoop how things can work for them.



[1]: http://getbootstrap.com/
[2]: http://www.apache.org/dev/release.html
[3]: https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin

-- 
Sean

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message