hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Dimiduk <ndimi...@gmail.com>
Subject Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)
Date Mon, 22 Jun 2015 20:03:53 GMT
On Mon, Jun 22, 2015 at 12:43 PM, Colin P. McCabe <cmccabe@apache.org>
wrote:

> You mentioned that "most of our project will be focused on shell
> scripts" I guess based on the existing test-patch code.  Allen did a
> lot of good work in this area recently.  I am curious if you evaluated
> languages such as Python or Node.js for this use-case.  Shell scripts
> can get a little... tricky beyond a certain size.  On the other hand,
> if we are standardizing on shell, which shell and which version?
> Perhaps bash 3.5+?
>

I'll also add that shell is not helpful for a cross-platform set of
tooling. I recently added a daemon to Apache Phoenix; an explicit
requirement was Windows support. I ended up implementing a solution in
python because that environment is platform-agnostic and still systems-y
enough. I think this is something this project should seriously consider.

-n

On Tue, Jun 16, 2015 at 7:55 PM, Sean Busbey <busbey@cloudera.com> wrote:
> > I'm going to try responding to several things at once here, so apologies
> if
> > I miss anyone and sorry for the long email. :)
> >
> >
> > On Tue, Jun 16, 2015 at 3:44 PM, Steve Loughran <stevel@hortonworks.com>
> > wrote:
> >
> >> I think it's good to have a general build/test process projects can
> share,
> >> so +1 to pulling it out. You should get help from others.
> >>
> >> regarding incubation, it is a lot of work, especially for something
> that's
> >> more of an in-house tool than an artifact to release and redistribute.
> >>
> >> You can't just use apache labs or the build project's repo to work on
> this?
> >>
> >> if you do want to incubate, we may want to nominate the hadoop project
> as
> >> the monitoring PMC, rather than incubator@.
> >>
> >> -steve
> >>
> >>
> > Important note: we're proposing a board resolution that would directly
> pull
> > this code base out into a new TLP; there'd be no incubator, we'd just
> > continue building community and start making releases.
> >
> > The proposed PMC believes the tooling we're talking about has direct
> > applicability to projects well outside of the ASF. Lot's of other open
> > source projects run on community contributions and have a general need
> for
> > better QA tools. Given that problem set and the presence of a community
> > working to solve it, there's no reason this needs to be treated as an
> > in-house build project. We certainly want to be useful to ASF projects
> and
> > getting them on-board given our current optimization for ASF infra will
> > certainly be easier, but we're not limited to that (and our current
> > prerequisites, a CI tool and jira or github, are pretty broadly
> available).
> >
> >
> > On Tue, Jun 16, 2015 at 10:13 AM, Nick Dimiduk <ndimiduk@apache.org>
> wrote:
> >
> >>
> >> Since we're tossing out names, how about Apache Bootstrap? It's a
> >> meta-project to help other projects get off the ground, after all.
> >>
> >
> >
> > There's already a web development framework named Bootstrap[1]. It's also
> > used by several ASF projects, so I think it best to avoid the confusion.
> >
> > The name is, of course, up to the proposed PMC. As a bit of background,
> the
> > current name Yetus fulfills Allen's desire to have something shell
> related
> > and my desire to have a project that starts with Y (there are currently
> no
> > ASF projects that start with Y). The universe of names that fill in these
> > two is very small, AFAICT. I did a brief suitability search and didn't
> find
> > any blockers.
> >
> >
> >  On Tue, Jun 16, 2015 at 11:59 AM, Allen Wittenauer <aw@altiscale.com>
> >  wrote:
> >
> >>
> >> Since a couple of people have brought it up:
> >>
> >>         I think the release question is probably one of the big question
> >> marks.  Other than tar balls, how does something like this actually get
> >> used downstream?
> >>
> >>         For test-patch, in particular, I have a few thoughts on this:
> >>
> >> Short term:
> >>
> >>         * Projects that want to move RIGHT NOW would modify their
> Jenkins
> >> jobs to checkout from the Yetus repo (preferably at a well known tag or
> >> branch) in one directory and their project repo in another directory.
> Then
> >> it’s just a matter of passing the correct flags to test-patch.  This is
> >> pretty much how I’ve been personally running test-patch for about 6
> months
> >> now. Under Jenkins, we’ve seen this work with NiFi (incubating) already.
> >>
> >>         * Create a stub version of test-patch that projects could check
> >> into their repo, replacing the existing test-patch.  This stub version
> >> would git clone from either ASF or github and then execute test-patch
> >> accordingly on demand.  With the correct smarts, it could make sure it
> has
> >> a cached version to prevent continual clones.
> >>
> >> Longer term:
> >>
> >>         * I’ve been toying with the idea of (ab)using Java repos and
> >> packaging as a transportation layer, either in addition or in
> combination
> >> with something like a maven plugin.  Something like this would clearly
> be
> >> better for offline usage and/or to lower the network traffic.
> >>
> >
> > It's important that the project follow ASF guidelines on publishing
> > releases[2]. So long as we publish releases to the distribution
> directory I
> > think we'd be fine having folks work off of the corresponding tag. I'm
> not
> > sure there's much reason to do that, however. A Jenkins job can just as
> > easily grab a release tarball as a git tag and we're not talking about a
> > large amount of stuff. The kind of build setup that Chris N mentioned is
> > also totally doable now that there's a build description DSL for
> Jenkins[3].
> >
> > For individual developers, I don't see any reason we can't package things
> > up as a tool, similar to how findbugs or shellcheck work. We can make OS
> > packages (or homebrew for OS X) if we want to make stand alone
> installation
> > on developer machines real easy. Those same packages could be installed
> on
> > the ASF build machines, provided some ASF project wanted to make use of
> > Yetus.
> >
> > Having releases will incur some turn around time for when folks want to
> see
> > fixes, but that's a trade off around release cadence we can work out
> longer
> > term.
> >
> > I would like to have one or two projects that can work off of the
> bleeding
> > edge repo, but we'd have to get that to mesh with foundation policy. My
> gut
> > tells me we should be able to come up with an agreement that makes such a
> > project "part of the development community" but the specifics will have
> to
> > be worked out.
> >
> >
> > On Tue, Jun 16, 2015 at 4:41 AM, Jonathan Hsieh <jon@cloudera.com>
> wrote:
> >
> >> How would we have to add testing to our pre commit testing?
> >>
> >> Each project has likely customized their own core commit scripts
> (multiple
> >> Jvms versions, checkstyle, javadoc exceptions etc.). We should probably
> >> soliciit interest from other projects who already have fancy precommit
> >> tests beyond HBase/Hadoop too.
> >>
> >
> > I'm not sure if Allen's response above answered this for you. The current
> > state of test-patch has a plugin system for adding new tests. It is also
> > customizable to handle project idiosyncrasies (atleast the ones we've
> seen
> > so far, like unspecified module dependencies, multiple projects per repo,
> > or use of ant). Releases will naturally include docs on how to leverage
> > those to customize what a particular project needs tested pre-commit.
> >
> > Given the nature of the work Yetus is hoping to enable, I think it's safe
> > to assume the project community is going to be doing a fair bit of
> outreach
> > to help show projects outside of HBase/Hadoop how things can work for
> them.
> >
> >
> >
> > [1]: http://getbootstrap.com/
> > [2]: http://www.apache.org/dev/release.html
> > [3]: https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin
> >
> > --
> > Sean
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message