geode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Bretl <asf.mbr...@gmail.com>
Subject Re: [DISCUSS] CI improvements
Date Fri, 06 Oct 2017 18:51:36 GMT
Correct, there is no requirement from ASF about where to run CI or even to
run CI. I am all for the best tools and stable (and repeatable)
environments. I would be open to seeing how Concourse could work for Geode.

Once the Concourse environment is setup, probably would be best to update
the Jenkins job to a pipeline job type as well.

There a few questions I have, which can be discussed as we go:
- What does 'donate' mean for Pivotal? Is there a, for lack of a better
term, contract or is this a 'good faith' effort? I am hesitant to go to any
corporately controlled infrastructure, especially since the entity could
decide to stop funding.
- What types of activities will require admin assistance?
- Who will be watching the dev list for requests?

--Mark

On Fri, Oct 6, 2017 at 10:05 AM, Kenneth Howe <khowe@pivotal.io> wrote:

> +1 - Reduce the noise level for analyzing CI results
>
> > On Oct 6, 2017, at 9:49 AM, Udo Kohlmeyer <ukohlmeyer@pivotal.io> wrote:
> >
> > +1 Switch... parallel runs would be safest
> >
> > On Fri, Oct 6, 2017 at 9:42 AM, Jianxia Chen <jchen21@apache.org> wrote:
> >
> >> +1 to switch to Concourse
> >> +1 As a first step we could run both CI systems side-by-side and see how
> >> the Concourse approach works for our project.
> >>
> >> On Fri, Oct 6, 2017 at 7:08 AM, Anthony Baker <abaker@pivotal.io>
> wrote:
> >>
> >>> Hi all,
> >>>
> >>> I’d like to propose the following that we switch our continuous
> >>> integration (CI) system from Jenkins [1] to Concourse [2].  I suggest
> >>> this because we continue to experience a significant number of
> >>> environmental-related test failures.
> >>>
> >>> These issues include CPU interference from other Jenkins jobs on the
> >>> same host, running out of disk space, port conflicts, and other
> >>> gremlins.  The net effect is that we are only getting 1-2 successful
> >>> builds per month.  Certainly not all test failures can be traced back
> >>> to environmental issues.  However, internal testing on isolated VM’s
> >>> shows a combined success rate of about 3X higher compared to ASF
> >>> Jenkins for the same tests.  This is still definitely NotAwesome, but
> >>> removing environmental factors will let us focus on stabilizing flaky
> >>> tests.
> >>>
> >>> Concourse is an Apache-licensed open source CI system based on
> >>> pipelines.  The pipelines are defined in a YML file containing job
> >>> definitions—inputs, outputs, resources, and tasks.  A task is simply a
> >>> bash script that returns 0/1 for success/failure.  A web UI displays
> >>> build status.  Importantly, each job runs inside an isolated
> >>> container.  The containers are load-balanced across a pool of workers.
> >>> For an example of a build pipeline, see [3] for the pipeline used to
> >>> build concourse itself.
> >>>
> >>> A Concourse environment is deployed and managed in cloud environments
> >>> through bosh [4].  Pivotal has agreed to donate AWS and/or GCP compute
> >>> and storage resources as well as manage the infrastructure.  These
> >>> project resources would be available for use by all committers and
> >>> community members regardless of corporate affiliations.  Note that
> >>> AFAIK there is no explicit requirement to host CI on ASF
> >>> infrastructure—unlike for critical project resources such as source
> >>> code, mailing lists, and issue tracking.
> >>>
> >>> The source for the pipeline and job scripts would reside within the
> >>> geode-* repos.  Geode committers would be able to modify those, same
> >>> as with our .travis.yml scripts.  All test results and build artifacts
> >>> would be publicly viewable just like with our Jenkins build output
> >>> today.  Requests for admin assistance would go through the dev@geode
> >>> mailing list.
> >>>
> >>> Thoughts?  As a first step we could run both CI systems side-by-side
> >>> and see how the Concourse approach works for our project.
> >>>
> >>> Thanks,
> >>> Anthony
> >>>
> >>>
> >>> [1] https://builds.apache.org/job/Geode-nightly/
> >>> [2] https://concourse.ci
> >>> [3] https://ci.concourse.ci
> >>> [4] https://bosh.io
> >>>
> >>
> >
> >
> >
> > --
> > Kindest Regards
> > -----------------------------
> > *Udo Kohlmeyer* | *Snr Solutions Architect* |*Pivotal*
> > *Mobile:* +61 409-279-160 | ukohlmeyer@pivotal.io
> > <http://www.gopivotal.com/>
> > www.pivotal.io
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message