geode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kirk Lund <kl...@apache.org>
Subject Re: [DISCUSS] CI improvements
Date Fri, 06 Oct 2017 16:26:02 GMT
+1 no thoughts other than make it so!

On Fri, Oct 6, 2017 at 7:08 AM, Anthony Baker <abaker@pivotal.io> wrote:

> Hi all,
>
> I’d like to propose the following that we switch our continuous
> integration (CI) system from Jenkins [1] to Concourse [2].  I suggest
> this because we continue to experience a significant number of
> environmental-related test failures.
>
> These issues include CPU interference from other Jenkins jobs on the
> same host, running out of disk space, port conflicts, and other
> gremlins.  The net effect is that we are only getting 1-2 successful
> builds per month.  Certainly not all test failures can be traced back
> to environmental issues.  However, internal testing on isolated VM’s
> shows a combined success rate of about 3X higher compared to ASF
> Jenkins for the same tests.  This is still definitely NotAwesome, but
> removing environmental factors will let us focus on stabilizing flaky
> tests.
>
> Concourse is an Apache-licensed open source CI system based on
> pipelines.  The pipelines are defined in a YML file containing job
> definitions—inputs, outputs, resources, and tasks.  A task is simply a
> bash script that returns 0/1 for success/failure.  A web UI displays
> build status.  Importantly, each job runs inside an isolated
> container.  The containers are load-balanced across a pool of workers.
> For an example of a build pipeline, see [3] for the pipeline used to
> build concourse itself.
>
> A Concourse environment is deployed and managed in cloud environments
> through bosh [4].  Pivotal has agreed to donate AWS and/or GCP compute
> and storage resources as well as manage the infrastructure.  These
> project resources would be available for use by all committers and
> community members regardless of corporate affiliations.  Note that
> AFAIK there is no explicit requirement to host CI on ASF
> infrastructure—unlike for critical project resources such as source
> code, mailing lists, and issue tracking.
>
> The source for the pipeline and job scripts would reside within the
> geode-* repos.  Geode committers would be able to modify those, same
> as with our .travis.yml scripts.  All test results and build artifacts
> would be publicly viewable just like with our Jenkins build output
> today.  Requests for admin assistance would go through the dev@geode
> mailing list.
>
> Thoughts?  As a first step we could run both CI systems side-by-side
> and see how the Concourse approach works for our project.
>
> Thanks,
> Anthony
>
>
> [1] https://builds.apache.org/job/Geode-nightly/
> [2] https://concourse.ci
> [3] https://ci.concourse.ci
> [4] https://bosh.io
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message