crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <jwi...@cloudera.com>
Subject Re: Continuous Integration: Next Steps
Date Mon, 06 Aug 2012 00:42:34 GMT
Ya know, Matthias understands this stuff. Brock, can you give Matthias
(mafr) admin access to Jenkins?

J

On Sun, Aug 5, 2012 at 5:23 PM, Josh Wills <jwills@cloudera.com> wrote:

> Thanks Brock. I edited it to become incubator-crunch, and it's running
> now: https://builds.apache.org/job/incubator-crunch/
>
> My Jenkins knowledge is also near-zero, so if anyone has any settings they
> would like me to enable, please let me know.
>
> On Thu, Aug 2, 2012 at 7:14 AM, Brock Noland <brock@cloudera.com> wrote:
>
>> Hmm. My experience with Jenkins is nearly zero.  I just created
>> "crunch-trunk" job, can you edit it?
>>
>> https://builds.apache.org/job/crunch-trunk/configure
>>
>>
>>
>> On Thu, Aug 2, 2012 at 9:03 AM, Josh Wills <josh.wills@gmail.com> wrote:
>>
>> > I get "jwills is missing the Create permission"
>> >
>> > On Thu, Aug 2, 2012 at 6:49 AM, Brock Noland <brock@cloudera.com>
>> wrote:
>> > > According to ldap, we are both in the "hudson-jobadmin" group. There
>> is
>> > no
>> > > "New Job" link in the upper left? The link goes here:
>> > > https://builds.apache.org/view/All/newJob
>> > >
>> > > Brock
>> > >
>> > > On Thu, Aug 2, 2012 at 12:13 AM, Josh Wills <jwills@cloudera.com>
>> wrote:
>> > >
>> > >> Hey Brock,
>> > >>
>> > >> I can log in to Jenkins at builds.apache.org, but it doesn't appear
>> > >> that I can create a new job-- I just have a dashboard that lists
>> Build
>> > >> History/Project Relationship/etc. Is there some admin bit I'm
>> missing?
>> > >>
>> > >> J
>> > >>
>> > >> On Tue, Jul 31, 2012 at 7:40 AM, Brock Noland <brock@cloudera.com>
>> > wrote:
>> > >> > Done!
>> > >> >
>> > >> > Brock
>> > >> >
>> > >> > On Tue, Jul 31, 2012 at 9:36 AM, Josh Wills <jwills@cloudera.com>
>> > wrote:
>> > >> >> Explicitly emailing Brock, Tom, and Arun, who I believe are
all of
>> > the
>> > >> PMC
>> > >> >> chairs on our teams-- guys, could one of your grant me (jwills)
>> > Jenkins
>> > >> >> access so I can setup CI?
>> > >> >>
>> > >> >> J
>> > >> >>
>> > >> >>
>> > >> >> On Mon, Jul 30, 2012 at 12:59 AM, Rahul <rsharma@xebia.com>
>> wrote:
>> > >> >>>
>> > >> >>> I think sonar will be a good way forward.
>> > >> >>>
>> > >> >>> As for maven repo for snapshots, there is one snapshots
repo from
>> > >> Apache
>> > >> >>> at https://repository.apache.org/content/repositories/snapshots/
>> .
>> > >> >>> I think this is good enough for the purpose and released
>> artifacts
>> > can
>> > >> be
>> > >> >>> uploaded to maven repos.
>> > >> >>>
>> > >> >>> regards
>> > >> >>> Rahul
>> > >> >>>
>> > >> >>>
>> > >> >>> On 29-07-2012 13:32, Matthias Friedrich wrote:
>> > >> >>>>
>> > >> >>>> Hi,
>> > >> >>>>
>> > >> >>>> over the last three weeks we've been working hard
on making
>> Crunch
>> > >> >>>> ready for CI. It turned out to be much more work than
initially
>> > >> >>>> expected and I cursed myself more than once for starting
this.
>> In
>> > the
>> > >> >>>> end it took four people and many patches to pull this
off
>> (thanks a
>> > >> >>>> lot for your help, Josh, Rahul, and Gabriel!). In
case you're
>> > >> >>>> interested, these were the issues that lead us to
CI readiness:
>> > >> >>>>
>> > >> >>>>   * CRUNCH-8: Refactor Crunch's build into a multi-module
Maven
>> > >> project
>> > >> >>>>   * CRUNCH-17: Split unit tests from integration tests
>> > >> >>>>   * CRUNCH-18: Improve build stability
>> > >> >>>>   * CRUNCH-21: Make temporary directory configurable
>> > >> >>>>   * CRUNCH-24: Make test suite suiteable for continuous
>> integration
>> > >> >>>>   * CRUNCH-26: Make Scrunch tests suiteable for continuous
>> > integration
>> > >> >>>>
>> > >> >>>> We can now start with basic jobs to detect broken
commits by
>> > running
>> > >> >>>> the unit and integration test suites. I suggest starting
with an
>> > >> >>>> on-commit build using "mvn package" and a nightly
build using
>> "mvn
>> > >> >>>> verify" (which runs the integration test suite). Running
the
>> > >> >>>> integration tests on-commit would be great but may
be too
>> > expensive.
>> > >> >>>>
>> > >> >>>> To get started we need access to Apache's Jenkins
instance [1]
>> > which
>> > >> >>>> apparently only PMC chairs can grant [2]. I'm new
to Apache
>> > processes,
>> > >> >>>> but perhaps a mentor can help us out here?
>> > >> >>>>
>> > >> >>>>
>> > >> >>>> Next steps:
>> > >> >>>>
>> > >> >>>> CI gives us the ability to regularly run reports to
check code
>> > >> >>>> quality. We already started preparations for this
(Maven site
>> > setup,
>> > >> >>>> checkstyle config and basic reports), but I think
the best thing
>> > would
>> > >> >>>> be to use Apache's Sonar instance [3]. I use Sonar
a lot at
>> work,
>> > it
>> > >> >>>> bundles all kinds of code metrics in a nice interface
and points
>> > out
>> > >> >>>> where refactoring time is best spent. It's the most
convenient
>> way
>> > of
>> > >> >>>> tracking code quality over time and catches many issues
human
>> > >> >>>> reviewers usually overlook.
>> > >> >>>>
>> > >> >>>> Additional to reports, it's also quite common to deploy
nightly
>> > >> >>>> SNAPSHOT builds to a Maven repository and to publish
up-to-date
>> > Maven
>> > >> >>>> sites. I'm not sure if this is possible at Apache
for legal
>> reasons
>> > >> >>>> though.
>> > >> >>>>
>> > >> >>>>
>> > >> >>>> Please let me know where you think we should go next.
I know
>> this
>> > >> >>>> whole CI episode has been a nuisance, so thanks again
for your
>> > help.
>> > >> >>>>
>> > >> >>>> Regards,
>> > >> >>>>    Matthias
>> > >> >>>>
>> > >> >>>> [1] https://builds.apache.org
>> > >> >>>> [2] http://wiki.apache.org/general/Jenkins
>> > >> >>>> [3] https://analysis.apache.org
>> > >> >>>
>> > >> >>>
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >> --
>> > >> >> Director of Data Science
>> > >> >> Cloudera
>> > >> >> Twitter: @josh_wills
>> > >> >>
>> > >> >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > Apache MRUnit - Unit testing MapReduce -
>> > >> http://incubator.apache.org/mrunit/
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> Director of Data Science
>> > >> Cloudera
>> > >> Twitter: @josh_wills
>> > >>
>> > >
>> > >
>> > >
>> > > --
>> > > Apache MRUnit - Unit testing MapReduce -
>> > http://incubator.apache.org/mrunit/
>> >
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>>
>
>
>
> --
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>
>


-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message