geode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jinmei Liao <jil...@pivotal.io>
Subject Re: [DISCUSS] CI improvements
Date Fri, 06 Oct 2017 15:52:19 GMT
+1 for switching concourse and running it side by side first to see how it
works first.

On Fri, Oct 6, 2017 at 7:57 AM, Gregory Chase <gchase@pivotal.io> wrote:

> I recall hearing that Apache HAWQ already runs Concourse in a similar
> approach.
>
> I also know of at least three other open source projects are running
> Concourse:
>
> 1. Concourse.ci <https://ci.concourse.ci/> itself
> 2. Greenplum Database <http://gpdb.ci.pivotalci.info/>
> 3. RabbitMQ <https://ci-aws.rabbitmq.com/>
>
> On Fri, Oct 6, 2017 at 2:08 PM, Anthony Baker <abaker@pivotal.io> wrote:
>
> > Hi all,
> >
> > I’d like to propose the following that we switch our continuous
> > integration (CI) system from Jenkins [1] to Concourse [2].  I suggest
> > this because we continue to experience a significant number of
> > environmental-related test failures.
> >
> > These issues include CPU interference from other Jenkins jobs on the
> > same host, running out of disk space, port conflicts, and other
> > gremlins.  The net effect is that we are only getting 1-2 successful
> > builds per month.  Certainly not all test failures can be traced back
> > to environmental issues.  However, internal testing on isolated VM’s
> > shows a combined success rate of about 3X higher compared to ASF
> > Jenkins for the same tests.  This is still definitely NotAwesome, but
> > removing environmental factors will let us focus on stabilizing flaky
> > tests.
> >
> > Concourse is an Apache-licensed open source CI system based on
> > pipelines.  The pipelines are defined in a YML file containing job
> > definitions—inputs, outputs, resources, and tasks.  A task is simply a
> > bash script that returns 0/1 for success/failure.  A web UI displays
> > build status.  Importantly, each job runs inside an isolated
> > container.  The containers are load-balanced across a pool of workers.
> > For an example of a build pipeline, see [3] for the pipeline used to
> > build concourse itself.
> >
> > A Concourse environment is deployed and managed in cloud environments
> > through bosh [4].  Pivotal has agreed to donate AWS and/or GCP compute
> > and storage resources as well as manage the infrastructure.  These
> > project resources would be available for use by all committers and
> > community members regardless of corporate affiliations.  Note that
> > AFAIK there is no explicit requirement to host CI on ASF
> > infrastructure—unlike for critical project resources such as source
> > code, mailing lists, and issue tracking.
> >
> > The source for the pipeline and job scripts would reside within the
> > geode-* repos.  Geode committers would be able to modify those, same
> > as with our .travis.yml scripts.  All test results and build artifacts
> > would be publicly viewable just like with our Jenkins build output
> > today.  Requests for admin assistance would go through the dev@geode
> > mailing list.
> >
> > Thoughts?  As a first step we could run both CI systems side-by-side
> > and see how the Concourse approach works for our project.
> >
> > Thanks,
> > Anthony
> >
> >
> > [1] https://builds.apache.org/job/Geode-nightly/
> > [2] https://concourse.ci
> > [3] https://ci.concourse.ci
> > [4] https://bosh.io
> >
>
>
>
> --
> Greg Chase
>
> Product team, Pivotal Cloud Foundry Services
> https://pivotal.io/platform/services
>
> Pivotal Software
> http://www.pivotal.io/
>
> 650-215-0477
> @GregChase
>



-- 
Cheers

Jinmei

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message