geode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gregory Chase <gch...@pivotal.io>
Subject Re: [DISCUSS] CI improvements
Date Fri, 06 Oct 2017 14:57:05 GMT
I recall hearing that Apache HAWQ already runs Concourse in a similar
approach.

I also know of at least three other open source projects are running
Concourse:

1. Concourse.ci <https://ci.concourse.ci/> itself
2. Greenplum Database <http://gpdb.ci.pivotalci.info/>
3. RabbitMQ <https://ci-aws.rabbitmq.com/>

On Fri, Oct 6, 2017 at 2:08 PM, Anthony Baker <abaker@pivotal.io> wrote:

> Hi all,
>
> I’d like to propose the following that we switch our continuous
> integration (CI) system from Jenkins [1] to Concourse [2].  I suggest
> this because we continue to experience a significant number of
> environmental-related test failures.
>
> These issues include CPU interference from other Jenkins jobs on the
> same host, running out of disk space, port conflicts, and other
> gremlins.  The net effect is that we are only getting 1-2 successful
> builds per month.  Certainly not all test failures can be traced back
> to environmental issues.  However, internal testing on isolated VM’s
> shows a combined success rate of about 3X higher compared to ASF
> Jenkins for the same tests.  This is still definitely NotAwesome, but
> removing environmental factors will let us focus on stabilizing flaky
> tests.
>
> Concourse is an Apache-licensed open source CI system based on
> pipelines.  The pipelines are defined in a YML file containing job
> definitions—inputs, outputs, resources, and tasks.  A task is simply a
> bash script that returns 0/1 for success/failure.  A web UI displays
> build status.  Importantly, each job runs inside an isolated
> container.  The containers are load-balanced across a pool of workers.
> For an example of a build pipeline, see [3] for the pipeline used to
> build concourse itself.
>
> A Concourse environment is deployed and managed in cloud environments
> through bosh [4].  Pivotal has agreed to donate AWS and/or GCP compute
> and storage resources as well as manage the infrastructure.  These
> project resources would be available for use by all committers and
> community members regardless of corporate affiliations.  Note that
> AFAIK there is no explicit requirement to host CI on ASF
> infrastructure—unlike for critical project resources such as source
> code, mailing lists, and issue tracking.
>
> The source for the pipeline and job scripts would reside within the
> geode-* repos.  Geode committers would be able to modify those, same
> as with our .travis.yml scripts.  All test results and build artifacts
> would be publicly viewable just like with our Jenkins build output
> today.  Requests for admin assistance would go through the dev@geode
> mailing list.
>
> Thoughts?  As a first step we could run both CI systems side-by-side
> and see how the Concourse approach works for our project.
>
> Thanks,
> Anthony
>
>
> [1] https://builds.apache.org/job/Geode-nightly/
> [2] https://concourse.ci
> [3] https://ci.concourse.ci
> [4] https://bosh.io
>



-- 
Greg Chase

Product team, Pivotal Cloud Foundry Services
https://pivotal.io/platform/services

Pivotal Software
http://www.pivotal.io/

650-215-0477
@GregChase

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message