geode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jared Stewart <jstew...@pivotal.io>
Subject Re: [DISCUSS] CI improvements
Date Fri, 06 Oct 2017 16:30:22 GMT
+1 I think this will be a huge improvement to the reliability of our test infrastructure.

- Jared

> On Oct 6, 2017, at 9:26 AM, Kirk Lund <klund@apache.org> wrote:
> 
> +1 no thoughts other than make it so!
> 
> On Fri, Oct 6, 2017 at 7:08 AM, Anthony Baker <abaker@pivotal.io> wrote:
> 
>> Hi all,
>> 
>> I’d like to propose the following that we switch our continuous
>> integration (CI) system from Jenkins [1] to Concourse [2].  I suggest
>> this because we continue to experience a significant number of
>> environmental-related test failures.
>> 
>> These issues include CPU interference from other Jenkins jobs on the
>> same host, running out of disk space, port conflicts, and other
>> gremlins.  The net effect is that we are only getting 1-2 successful
>> builds per month.  Certainly not all test failures can be traced back
>> to environmental issues.  However, internal testing on isolated VM’s
>> shows a combined success rate of about 3X higher compared to ASF
>> Jenkins for the same tests.  This is still definitely NotAwesome, but
>> removing environmental factors will let us focus on stabilizing flaky
>> tests.
>> 
>> Concourse is an Apache-licensed open source CI system based on
>> pipelines.  The pipelines are defined in a YML file containing job
>> definitions—inputs, outputs, resources, and tasks.  A task is simply a
>> bash script that returns 0/1 for success/failure.  A web UI displays
>> build status.  Importantly, each job runs inside an isolated
>> container.  The containers are load-balanced across a pool of workers.
>> For an example of a build pipeline, see [3] for the pipeline used to
>> build concourse itself.
>> 
>> A Concourse environment is deployed and managed in cloud environments
>> through bosh [4].  Pivotal has agreed to donate AWS and/or GCP compute
>> and storage resources as well as manage the infrastructure.  These
>> project resources would be available for use by all committers and
>> community members regardless of corporate affiliations.  Note that
>> AFAIK there is no explicit requirement to host CI on ASF
>> infrastructure—unlike for critical project resources such as source
>> code, mailing lists, and issue tracking.
>> 
>> The source for the pipeline and job scripts would reside within the
>> geode-* repos.  Geode committers would be able to modify those, same
>> as with our .travis.yml scripts.  All test results and build artifacts
>> would be publicly viewable just like with our Jenkins build output
>> today.  Requests for admin assistance would go through the dev@geode
>> mailing list.
>> 
>> Thoughts?  As a first step we could run both CI systems side-by-side
>> and see how the Concourse approach works for our project.
>> 
>> Thanks,
>> Anthony
>> 
>> 
>> [1] https://builds.apache.org/job/Geode-nightly/
>> [2] https://concourse.ci
>> [3] https://ci.concourse.ci
>> [4] https://bosh.io
>> 


Mime
View raw message