Return-Path: X-Original-To: apmail-hbase-dev-archive@www.apache.org Delivered-To: apmail-hbase-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E6322185BC for ; Mon, 22 Jun 2015 20:55:59 +0000 (UTC) Received: (qmail 7169 invoked by uid 500); 22 Jun 2015 20:55:57 -0000 Delivered-To: apmail-hbase-dev-archive@hbase.apache.org Received: (qmail 7060 invoked by uid 500); 22 Jun 2015 20:55:57 -0000 Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list dev@hbase.apache.org Received: (qmail 7039 invoked by uid 99); 22 Jun 2015 20:55:57 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Jun 2015 20:55:56 +0000 Received: from mail-oi0-f44.google.com (mail-oi0-f44.google.com [209.85.218.44]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id A30361A0323 for ; Mon, 22 Jun 2015 20:55:56 +0000 (UTC) Received: by oigx81 with SMTP id x81so130908385oig.1 for ; Mon, 22 Jun 2015 13:55:55 -0700 (PDT) X-Received: by 10.202.168.85 with SMTP id r82mr25100513oie.55.1435006555709; Mon, 22 Jun 2015 13:55:55 -0700 (PDT) MIME-Version: 1.0 Received: by 10.202.83.20 with HTTP; Mon, 22 Jun 2015 13:55:16 -0700 (PDT) In-Reply-To: References: <2C3CADA4-3F36-4D47-8008-43CE80E28D38@altiscale.com> From: Andrew Purtell Date: Mon, 22 Jun 2015 13:55:16 -0700 Message-ID: Subject: Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?) To: "common-dev@hadoop.apache.org" Cc: hbase-dev Content-Type: multipart/alternative; boundary=001a113ce34c0125900519218001 --001a113ce34c0125900519218001 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Mon, Jun 22, 2015 at 1:03 PM, Nick Dimiduk wrote: > On Mon, Jun 22, 2015 at 12:43 PM, Colin P. McCabe > wrote: > > > You mentioned that "most of our project will be focused on shell > > scripts" I guess based on the existing test-patch code. Allen did a > > lot of good work in this area recently. I am curious if you evaluated > > languages such as Python or Node.js for this use-case. Shell scripts > > can get a little... tricky beyond a certain size. On the other hand, > > if we are standardizing on shell, which shell and which version? > > Perhaps bash 3.5+? > > > > I'll also add that shell is not helpful for a cross-platform set of > tooling. I recently added a daemon to Apache Phoenix; an explicit > requirement was Windows support. I ended up implementing a solution in > python because that environment is platform-agnostic and still systems-y > enough. I think this is something this project should seriously consider. > In my opinion, historically, test-patch hasn't needed to be cross platform because the only first class development environment for Hadoop has been Linux. Growing beyond this could absolutely be one focus of Yetus should that be a consensus goal of the community. The seed of the project, though, is today's test-patch, which is implemented in bash. That's where we are today. Language "discussions" (smile) can and should be forward looking. On Mon, Jun 22, 2015 at 1:03 PM, Nick Dimiduk wrote: > On Mon, Jun 22, 2015 at 12:43 PM, Colin P. McCabe > wrote: > > > You mentioned that "most of our project will be focused on shell > > scripts" I guess based on the existing test-patch code. Allen did a > > lot of good work in this area recently. I am curious if you evaluated > > languages such as Python or Node.js for this use-case. Shell scripts > > can get a little... tricky beyond a certain size. On the other hand, > > if we are standardizing on shell, which shell and which version? > > Perhaps bash 3.5+? > > > > I'll also add that shell is not helpful for a cross-platform set of > tooling. I recently added a daemon to Apache Phoenix; an explicit > requirement was Windows support. I ended up implementing a solution in > python because that environment is platform-agnostic and still systems-y > enough. I think this is something this project should seriously consider. > > -n > > On Tue, Jun 16, 2015 at 7:55 PM, Sean Busbey wrote: > > > I'm going to try responding to several things at once here, so > apologies > > if > > > I miss anyone and sorry for the long email. :) > > > > > > > > > On Tue, Jun 16, 2015 at 3:44 PM, Steve Loughran < > stevel@hortonworks.com> > > > wrote: > > > > > >> I think it's good to have a general build/test process projects can > > share, > > >> so +1 to pulling it out. You should get help from others. > > >> > > >> regarding incubation, it is a lot of work, especially for something > > that's > > >> more of an in-house tool than an artifact to release and redistribut= e. > > >> > > >> You can't just use apache labs or the build project's repo to work o= n > > this? > > >> > > >> if you do want to incubate, we may want to nominate the hadoop proje= ct > > as > > >> the monitoring PMC, rather than incubator@. > > >> > > >> -steve > > >> > > >> > > > Important note: we're proposing a board resolution that would directl= y > > pull > > > this code base out into a new TLP; there'd be no incubator, we'd just > > > continue building community and start making releases. > > > > > > The proposed PMC believes the tooling we're talking about has direct > > > applicability to projects well outside of the ASF. Lot's of other ope= n > > > source projects run on community contributions and have a general nee= d > > for > > > better QA tools. Given that problem set and the presence of a communi= ty > > > working to solve it, there's no reason this needs to be treated as an > > > in-house build project. We certainly want to be useful to ASF project= s > > and > > > getting them on-board given our current optimization for ASF infra wi= ll > > > certainly be easier, but we're not limited to that (and our current > > > prerequisites, a CI tool and jira or github, are pretty broadly > > available). > > > > > > > > > On Tue, Jun 16, 2015 at 10:13 AM, Nick Dimiduk > > wrote: > > > > > >> > > >> Since we're tossing out names, how about Apache Bootstrap? It's a > > >> meta-project to help other projects get off the ground, after all. > > >> > > > > > > > > > There's already a web development framework named Bootstrap[1]. It's > also > > > used by several ASF projects, so I think it best to avoid the > confusion. > > > > > > The name is, of course, up to the proposed PMC. As a bit of backgroun= d, > > the > > > current name Yetus fulfills Allen's desire to have something shell > > related > > > and my desire to have a project that starts with Y (there are current= ly > > no > > > ASF projects that start with Y). The universe of names that fill in > these > > > two is very small, AFAICT. I did a brief suitability search and didn'= t > > find > > > any blockers. > > > > > > > > > On Tue, Jun 16, 2015 at 11:59 AM, Allen Wittenauer > > > wrote: > > > > > >> > > >> Since a couple of people have brought it up: > > >> > > >> I think the release question is probably one of the big > question > > >> marks. Other than tar balls, how does something like this actually > get > > >> used downstream? > > >> > > >> For test-patch, in particular, I have a few thoughts on this= : > > >> > > >> Short term: > > >> > > >> * Projects that want to move RIGHT NOW would modify their > > Jenkins > > >> jobs to checkout from the Yetus repo (preferably at a well known tag > or > > >> branch) in one directory and their project repo in another directory= . > > Then > > >> it=E2=80=99s just a matter of passing the correct flags to test-patc= h. This > is > > >> pretty much how I=E2=80=99ve been personally running test-patch for = about 6 > > months > > >> now. Under Jenkins, we=E2=80=99ve seen this work with NiFi (incubati= ng) > already. > > >> > > >> * Create a stub version of test-patch that projects could > check > > >> into their repo, replacing the existing test-patch. This stub versi= on > > >> would git clone from either ASF or github and then execute test-patc= h > > >> accordingly on demand. With the correct smarts, it could make sure = it > > has > > >> a cached version to prevent continual clones. > > >> > > >> Longer term: > > >> > > >> * I=E2=80=99ve been toying with the idea of (ab)using Java r= epos and > > >> packaging as a transportation layer, either in addition or in > > combination > > >> with something like a maven plugin. Something like this would clear= ly > > be > > >> better for offline usage and/or to lower the network traffic. > > >> > > > > > > It's important that the project follow ASF guidelines on publishing > > > releases[2]. So long as we publish releases to the distribution > > directory I > > > think we'd be fine having folks work off of the corresponding tag. I'= m > > not > > > sure there's much reason to do that, however. A Jenkins job can just = as > > > easily grab a release tarball as a git tag and we're not talking abou= t > a > > > large amount of stuff. The kind of build setup that Chris N mentioned > is > > > also totally doable now that there's a build description DSL for > > Jenkins[3]. > > > > > > For individual developers, I don't see any reason we can't package > things > > > up as a tool, similar to how findbugs or shellcheck work. We can make > OS > > > packages (or homebrew for OS X) if we want to make stand alone > > installation > > > on developer machines real easy. Those same packages could be install= ed > > on > > > the ASF build machines, provided some ASF project wanted to make use = of > > > Yetus. > > > > > > Having releases will incur some turn around time for when folks want = to > > see > > > fixes, but that's a trade off around release cadence we can work out > > longer > > > term. > > > > > > I would like to have one or two projects that can work off of the > > bleeding > > > edge repo, but we'd have to get that to mesh with foundation policy. = My > > gut > > > tells me we should be able to come up with an agreement that makes > such a > > > project "part of the development community" but the specifics will ha= ve > > to > > > be worked out. > > > > > > > > > On Tue, Jun 16, 2015 at 4:41 AM, Jonathan Hsieh > > wrote: > > > > > >> How would we have to add testing to our pre commit testing? > > >> > > >> Each project has likely customized their own core commit scripts > > (multiple > > >> Jvms versions, checkstyle, javadoc exceptions etc.). We should > probably > > >> soliciit interest from other projects who already have fancy precomm= it > > >> tests beyond HBase/Hadoop too. > > >> > > > > > > I'm not sure if Allen's response above answered this for you. The > current > > > state of test-patch has a plugin system for adding new tests. It is > also > > > customizable to handle project idiosyncrasies (atleast the ones we've > > seen > > > so far, like unspecified module dependencies, multiple projects per > repo, > > > or use of ant). Releases will naturally include docs on how to levera= ge > > > those to customize what a particular project needs tested pre-commit. > > > > > > Given the nature of the work Yetus is hoping to enable, I think it's > safe > > > to assume the project community is going to be doing a fair bit of > > outreach > > > to help show projects outside of HBase/Hadoop how things can work for > > them. > > > > > > > > > > > > [1]: http://getbootstrap.com/ > > > [2]: http://www.apache.org/dev/release.html > > > [3]: https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin > > > > > > -- > > > Sean > > > --=20 Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) --001a113ce34c0125900519218001--