geode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anthony Baker <aba...@pivotal.io>
Subject [DISCUSS] CI improvements
Date Fri, 06 Oct 2017 14:08:45 GMT
Hi all,

I’d like to propose the following that we switch our continuous
integration (CI) system from Jenkins [1] to Concourse [2].  I suggest
this because we continue to experience a significant number of
environmental-related test failures.

These issues include CPU interference from other Jenkins jobs on the
same host, running out of disk space, port conflicts, and other
gremlins.  The net effect is that we are only getting 1-2 successful
builds per month.  Certainly not all test failures can be traced back
to environmental issues.  However, internal testing on isolated VM’s
shows a combined success rate of about 3X higher compared to ASF
Jenkins for the same tests.  This is still definitely NotAwesome, but
removing environmental factors will let us focus on stabilizing flaky
tests.

Concourse is an Apache-licensed open source CI system based on
pipelines.  The pipelines are defined in a YML file containing job
definitions—inputs, outputs, resources, and tasks.  A task is simply a
bash script that returns 0/1 for success/failure.  A web UI displays
build status.  Importantly, each job runs inside an isolated
container.  The containers are load-balanced across a pool of workers.
For an example of a build pipeline, see [3] for the pipeline used to
build concourse itself.

A Concourse environment is deployed and managed in cloud environments
through bosh [4].  Pivotal has agreed to donate AWS and/or GCP compute
and storage resources as well as manage the infrastructure.  These
project resources would be available for use by all committers and
community members regardless of corporate affiliations.  Note that
AFAIK there is no explicit requirement to host CI on ASF
infrastructure—unlike for critical project resources such as source
code, mailing lists, and issue tracking.

The source for the pipeline and job scripts would reside within the
geode-* repos.  Geode committers would be able to modify those, same
as with our .travis.yml scripts.  All test results and build artifacts
would be publicly viewable just like with our Jenkins build output
today.  Requests for admin assistance would go through the dev@geode
mailing list.

Thoughts?  As a first step we could run both CI systems side-by-side
and see how the Concourse approach works for our project.

Thanks,
Anthony


[1] https://builds.apache.org/job/Geode-nightly/
[2] https://concourse.ci
[3] https://ci.concourse.ci
[4] https://bosh.io

Mime
View raw message