geode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Smith <dsm...@pivotal.io>
Subject Re: [DISCUSS] CI improvements
Date Thu, 02 Nov 2017 18:49:24 GMT
Looks good. Should we go ahead and change this to run precheckin instead of
build?

-Dan

On Thu, Nov 2, 2017 at 9:53 AM, Anthony Baker <abaker@pivotal.io> wrote:

> If you’d like to check this out, here’s the PR containing the pipeline and
> job scripts:
> https://github.com/apache/geode/pull/1006
>
> And the pipeline itself:
> https://concourse.apachegeode-ci.info
>
> There are three pipelines defined:
>
> - develop:  runs `gradle build`.  Can be extended to include other
> precheckin tests based on feedback.
> - docker-images: builds the container used for the develop pipeline.
> - meta: watches for changes to the pipeline files and automatically
> updates the runtime pipelines.
>
> Authentication is integrated with GitHub.  If you want the ability to
> manually stop/start jobs please request on the dev@g.a.o mailing list
> (same as for Jenkins) and include your GitHub id.
>
> What do you think?
>
> Anthony
>
> > On Oct 6, 2017, at 7:08 AM, Anthony Baker <abaker@pivotal.io> wrote:
> >
> > Hi all,
> >
> > I’d like to propose the following that we switch our continuous
> > integration (CI) system from Jenkins [1] to Concourse [2].  I suggest
> > this because we continue to experience a significant number of
> > environmental-related test failures.
> >
> > These issues include CPU interference from other Jenkins jobs on the
> > same host, running out of disk space, port conflicts, and other
> > gremlins.  The net effect is that we are only getting 1-2 successful
> > builds per month.  Certainly not all test failures can be traced back
> > to environmental issues.  However, internal testing on isolated VM’s
> > shows a combined success rate of about 3X higher compared to ASF
> > Jenkins for the same tests.  This is still definitely NotAwesome, but
> > removing environmental factors will let us focus on stabilizing flaky
> > tests.
> >
> > Concourse is an Apache-licensed open source CI system based on
> > pipelines.  The pipelines are defined in a YML file containing job
> > definitions—inputs, outputs, resources, and tasks.  A task is simply a
> > bash script that returns 0/1 for success/failure.  A web UI displays
> > build status.  Importantly, each job runs inside an isolated
> > container.  The containers are load-balanced across a pool of workers.
> > For an example of a build pipeline, see [3] for the pipeline used to
> > build concourse itself.
> >
> > A Concourse environment is deployed and managed in cloud environments
> > through bosh [4].  Pivotal has agreed to donate AWS and/or GCP compute
> > and storage resources as well as manage the infrastructure.  These
> > project resources would be available for use by all committers and
> > community members regardless of corporate affiliations.  Note that
> > AFAIK there is no explicit requirement to host CI on ASF
> > infrastructure—unlike for critical project resources such as source
> > code, mailing lists, and issue tracking.
> >
> > The source for the pipeline and job scripts would reside within the
> > geode-* repos.  Geode committers would be able to modify those, same
> > as with our .travis.yml scripts.  All test results and build artifacts
> > would be publicly viewable just like with our Jenkins build output
> > today.  Requests for admin assistance would go through the dev@geode
> > mailing list.
> >
> > Thoughts?  As a first step we could run both CI systems side-by-side
> > and see how the Concourse approach works for our project.
> >
> > Thanks,
> > Anthony
> >
> >
> > [1] https://builds.apache.org/job/Geode-nightly/
> > [2] https://concourse.ci
> > [3] https://ci.concourse.ci
> > [4] https://bosh.io
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message