Mailing-List: contact crunch-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: crunch-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of brock@cloudera.com designates
 209.85.220.175 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAH29n6PRqvSjyn1S2wwfEayij7KKZZ_hXE-CSZ7TS804ZOz-7A@mail.gmail.com>
References: <20120729080228.GA13917@mafr.de> <50163EE5.1010103@xebia.com>
 <CAH29n6O+LOM=hnymeiJMyNPPGgUeB5s2neoXxXnxvcx9OtxR2A@mail.gmail.com>
 <CAFukC=4YTL8HL-w9AoFmRv0Zq7QUfi5j1+LCUJ140YVsqVb0LA@mail.gmail.com>
 <CAH29n6M4schLxkmq14KHaFdqdpGE=nF8G_LTVLy1OkM8Wp95Qg@mail.gmail.com>
 <CAFukC=7SQt8yKTyM_JUHK8rW51WaGg+=KbUpiq5cvvcAH7GJZw@mail.gmail.com>
 <CANb5z2Jcp-7eBxuybdhUwZcp=fycF3aVca+N-aVkrOuVJNvXJQ@mail.gmail.com>
 <CAFukC=4jhm4=vnBDZyUAseMV_JZXx4NGcjJHaRxM_69xboRv+g@mail.gmail.com>
 <CAH29n6OP4B1EJ+Hwd9RE_rGSBzQW6PYZb16Zxboe6ms2ydYTcA@mail.gmail.com>
 <CAH29n6PRqvSjyn1S2wwfEayij7KKZZ_hXE-CSZ7TS804ZOz-7A@mail.gmail.com>
From: Brock Noland <brock@cloudera.com>
Date: Sun, 5 Aug 2012 20:40:38 -0500
Message-ID: 
 <CAFukC=5VDzZe7bUVAGv=kK32D7qbL0U+0NtyVopDRj4NkXODWg@mail.gmail.com>
Subject: Re: Continuous Integration: Next Steps
To: Josh Wills <jwills@cloudera.com>
Cc: crunch-dev@incubator.apache.org
Content-Type: multipart/alternative; boundary=047d7b6742382d188c04c68ef82c

--047d7b6742382d188c04c68ef82c
Content-Type: text/plain; charset=ISO-8859-1

done!

brock@minotaur:~$ list_appgroups.pl --user=mafr
hudson-jobadmin


On Sun, Aug 5, 2012 at 7:42 PM, Josh Wills <jwills@cloudera.com> wrote:

> Ya know, Matthias understands this stuff. Brock, can you give Matthias
> (mafr) admin access to Jenkins?
>
> J
>
> On Sun, Aug 5, 2012 at 5:23 PM, Josh Wills <jwills@cloudera.com> wrote:
>
>> Thanks Brock. I edited it to become incubator-crunch, and it's running
>> now: https://builds.apache.org/job/incubator-crunch/
>>
>> My Jenkins knowledge is also near-zero, so if anyone has any settings
>> they would like me to enable, please let me know.
>>
>> On Thu, Aug 2, 2012 at 7:14 AM, Brock Noland <brock@cloudera.com> wrote:
>>
>>> Hmm. My experience with Jenkins is nearly zero.  I just created
>>> "crunch-trunk" job, can you edit it?
>>>
>>> https://builds.apache.org/job/crunch-trunk/configure
>>>
>>>
>>>
>>> On Thu, Aug 2, 2012 at 9:03 AM, Josh Wills <josh.wills@gmail.com> wrote:
>>>
>>> > I get "jwills is missing the Create permission"
>>> >
>>> > On Thu, Aug 2, 2012 at 6:49 AM, Brock Noland <brock@cloudera.com>
>>> wrote:
>>> > > According to ldap, we are both in the "hudson-jobadmin" group. There
>>> is
>>> > no
>>> > > "New Job" link in the upper left? The link goes here:
>>> > > https://builds.apache.org/view/All/newJob
>>> > >
>>> > > Brock
>>> > >
>>> > > On Thu, Aug 2, 2012 at 12:13 AM, Josh Wills <jwills@cloudera.com>
>>> wrote:
>>> > >
>>> > >> Hey Brock,
>>> > >>
>>> > >> I can log in to Jenkins at builds.apache.org, but it doesn't appear
>>> > >> that I can create a new job-- I just have a dashboard that lists
>>> Build
>>> > >> History/Project Relationship/etc. Is there some admin bit I'm
>>> missing?
>>> > >>
>>> > >> J
>>> > >>
>>> > >> On Tue, Jul 31, 2012 at 7:40 AM, Brock Noland <brock@cloudera.com>
>>> > wrote:
>>> > >> > Done!
>>> > >> >
>>> > >> > Brock
>>> > >> >
>>> > >> > On Tue, Jul 31, 2012 at 9:36 AM, Josh Wills <jwills@cloudera.com>
>>> > wrote:
>>> > >> >> Explicitly emailing Brock, Tom, and Arun, who I believe are all
>>> of
>>> > the
>>> > >> PMC
>>> > >> >> chairs on our teams-- guys, could one of your grant me (jwills)
>>> > Jenkins
>>> > >> >> access so I can setup CI?
>>> > >> >>
>>> > >> >> J
>>> > >> >>
>>> > >> >>
>>> > >> >> On Mon, Jul 30, 2012 at 12:59 AM, Rahul <rsharma@xebia.com>
>>> wrote:
>>> > >> >>>
>>> > >> >>> I think sonar will be a good way forward.
>>> > >> >>>
>>> > >> >>> As for maven repo for snapshots, there is one snapshots repo
>>> from
>>> > >> Apache
>>> > >> >>> at
>>> https://repository.apache.org/content/repositories/snapshots/.
>>> > >> >>> I think this is good enough for the purpose and released
>>> artifacts
>>> > can
>>> > >> be
>>> > >> >>> uploaded to maven repos.
>>> > >> >>>
>>> > >> >>> regards
>>> > >> >>> Rahul
>>> > >> >>>
>>> > >> >>>
>>> > >> >>> On 29-07-2012 13:32, Matthias Friedrich wrote:
>>> > >> >>>>
>>> > >> >>>> Hi,
>>> > >> >>>>
>>> > >> >>>> over the last three weeks we've been working hard on making
>>> Crunch
>>> > >> >>>> ready for CI. It turned out to be much more work than initially
>>> > >> >>>> expected and I cursed myself more than once for starting this.
>>> In
>>> > the
>>> > >> >>>> end it took four people and many patches to pull this off
>>> (thanks a
>>> > >> >>>> lot for your help, Josh, Rahul, and Gabriel!). In case you're
>>> > >> >>>> interested, these were the issues that lead us to CI readiness:
>>> > >> >>>>
>>> > >> >>>>   * CRUNCH-8: Refactor Crunch's build into a multi-module Maven
>>> > >> project
>>> > >> >>>>   * CRUNCH-17: Split unit tests from integration tests
>>> > >> >>>>   * CRUNCH-18: Improve build stability
>>> > >> >>>>   * CRUNCH-21: Make temporary directory configurable
>>> > >> >>>>   * CRUNCH-24: Make test suite suiteable for continuous
>>> integration
>>> > >> >>>>   * CRUNCH-26: Make Scrunch tests suiteable for continuous
>>> > integration
>>> > >> >>>>
>>> > >> >>>> We can now start with basic jobs to detect broken commits by
>>> > running
>>> > >> >>>> the unit and integration test suites. I suggest starting with
>>> an
>>> > >> >>>> on-commit build using "mvn package" and a nightly build using
>>> "mvn
>>> > >> >>>> verify" (which runs the integration test suite). Running the
>>> > >> >>>> integration tests on-commit would be great but may be too
>>> > expensive.
>>> > >> >>>>
>>> > >> >>>> To get started we need access to Apache's Jenkins instance [1]
>>> > which
>>> > >> >>>> apparently only PMC chairs can grant [2]. I'm new to Apache
>>> > processes,
>>> > >> >>>> but perhaps a mentor can help us out here?
>>> > >> >>>>
>>> > >> >>>>
>>> > >> >>>> Next steps:
>>> > >> >>>>
>>> > >> >>>> CI gives us the ability to regularly run reports to check code
>>> > >> >>>> quality. We already started preparations for this (Maven site
>>> > setup,
>>> > >> >>>> checkstyle config and basic reports), but I think the best
>>> thing
>>> > would
>>> > >> >>>> be to use Apache's Sonar instance [3]. I use Sonar a lot at
>>> work,
>>> > it
>>> > >> >>>> bundles all kinds of code metrics in a nice interface and
>>> points
>>> > out
>>> > >> >>>> where refactoring time is best spent. It's the most convenient
>>> way
>>> > of
>>> > >> >>>> tracking code quality over time and catches many issues human
>>> > >> >>>> reviewers usually overlook.
>>> > >> >>>>
>>> > >> >>>> Additional to reports, it's also quite common to deploy nightly
>>> > >> >>>> SNAPSHOT builds to a Maven repository and to publish up-to-date
>>> > Maven
>>> > >> >>>> sites. I'm not sure if this is possible at Apache for legal
>>> reasons
>>> > >> >>>> though.
>>> > >> >>>>
>>> > >> >>>>
>>> > >> >>>> Please let me know where you think we should go next. I know
>>> this
>>> > >> >>>> whole CI episode has been a nuisance, so thanks again for your
>>> > help.
>>> > >> >>>>
>>> > >> >>>> Regards,
>>> > >> >>>>    Matthias
>>> > >> >>>>
>>> > >> >>>> [1] https://builds.apache.org
>>> > >> >>>> [2] http://wiki.apache.org/general/Jenkins
>>> > >> >>>> [3] https://analysis.apache.org
>>> > >> >>>
>>> > >> >>>
>>> > >> >>
>>> > >> >>
>>> > >> >>
>>> > >> >> --
>>> > >> >> Director of Data Science
>>> > >> >> Cloudera
>>> > >> >> Twitter: @josh_wills
>>> > >> >>
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > --
>>> > >> > Apache MRUnit - Unit testing MapReduce -
>>> > >> http://incubator.apache.org/mrunit/
>>> > >>
>>> > >>
>>> > >>
>>> > >> --
>>> > >> Director of Data Science
>>> > >> Cloudera
>>> > >> Twitter: @josh_wills
>>> > >>
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Apache MRUnit - Unit testing MapReduce -
>>> > http://incubator.apache.org/mrunit/
>>> >
>>>
>>>
>>>
>>> --
>>> Apache MRUnit - Unit testing MapReduce -
>>> http://incubator.apache.org/mrunit/
>>>
>>
>>
>>
>> --
>> Director of Data Science
>> Cloudera <http://www.cloudera.com>
>> Twitter: @josh_wills <http://twitter.com/josh_wills>
>>
>>
>
>
> --
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>
>


-- 
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

--047d7b6742382d188c04c68ef82c--