Mailing-List: contact dev-help@geode.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@geode.apache.org
MIME-Version: 1.0
In-Reply-To: <6E5EA8FA-73A2-432F-8479-553F9C490318@pivotal.io>
References: <CAA37xyeMtxorZA-dsSipUiVKNoLjnizfKKNqnEzumckhTd-HhQ@mail.gmail.com>
 <CAKR0ha=+sx+Gx48KQ23TWQD9x1dytf0LgDkz_zvc9-gi_VtDHw@mail.gmail.com> <6E5EA8FA-73A2-432F-8479-553F9C490318@pivotal.io>
From: Mark Bretl <asf.mbretl@gmail.com>
Date: Fri, 6 Oct 2017 11:51:36 -0700
Message-ID: <CA+OE_e0Uq6Urv_f=4U4HgisQ2=mAvBRQFcg0DbydAS=MABtzWw@mail.gmail.com>
Subject: Re: [DISCUSS] CI improvements
To: geode <dev@geode.apache.org>
Content-Type: multipart/alternative; boundary="94eb2c0724429ee009055ae554c2"
archived-at: Fri, 06 Oct 2017 18:51:44 -0000

--94eb2c0724429ee009055ae554c2
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Correct, there is no requirement from ASF about where to run CI or even to
run CI. I am all for the best tools and stable (and repeatable)
environments. I would be open to seeing how Concourse could work for Geode.

Once the Concourse environment is setup, probably would be best to update
the Jenkins job to a pipeline job type as well.

There a few questions I have, which can be discussed as we go:
- What does 'donate' mean for Pivotal? Is there a, for lack of a better
term, contract or is this a 'good faith' effort? I am hesitant to go to any
corporately controlled infrastructure, especially since the entity could
decide to stop funding.
- What types of activities will require admin assistance?
- Who will be watching the dev list for requests?

--Mark

On Fri, Oct 6, 2017 at 10:05 AM, Kenneth Howe <khowe@pivotal.io> wrote:

> +1 - Reduce the noise level for analyzing CI results
>
> > On Oct 6, 2017, at 9:49 AM, Udo Kohlmeyer <ukohlmeyer@pivotal.io> wrote=
:
> >
> > +1 Switch... parallel runs would be safest
> >
> > On Fri, Oct 6, 2017 at 9:42 AM, Jianxia Chen <jchen21@apache.org> wrote=
:
> >
> >> +1 to switch to Concourse
> >> +1 As a first step we could run both CI systems side-by-side and see h=
ow
> >> the Concourse approach works for our project.
> >>
> >> On Fri, Oct 6, 2017 at 7:08 AM, Anthony Baker <abaker@pivotal.io>
> wrote:
> >>
> >>> Hi all,
> >>>
> >>> I=E2=80=99d like to propose the following that we switch our continuo=
us
> >>> integration (CI) system from Jenkins [1] to Concourse [2].  I suggest
> >>> this because we continue to experience a significant number of
> >>> environmental-related test failures.
> >>>
> >>> These issues include CPU interference from other Jenkins jobs on the
> >>> same host, running out of disk space, port conflicts, and other
> >>> gremlins.  The net effect is that we are only getting 1-2 successful
> >>> builds per month.  Certainly not all test failures can be traced back
> >>> to environmental issues.  However, internal testing on isolated VM=E2=
=80=99s
> >>> shows a combined success rate of about 3X higher compared to ASF
> >>> Jenkins for the same tests.  This is still definitely NotAwesome, but
> >>> removing environmental factors will let us focus on stabilizing flaky
> >>> tests.
> >>>
> >>> Concourse is an Apache-licensed open source CI system based on
> >>> pipelines.  The pipelines are defined in a YML file containing job
> >>> definitions=E2=80=94inputs, outputs, resources, and tasks.  A task is=
 simply a
> >>> bash script that returns 0/1 for success/failure.  A web UI displays
> >>> build status.  Importantly, each job runs inside an isolated
> >>> container.  The containers are load-balanced across a pool of workers=
.
> >>> For an example of a build pipeline, see [3] for the pipeline used to
> >>> build concourse itself.
> >>>
> >>> A Concourse environment is deployed and managed in cloud environments
> >>> through bosh [4].  Pivotal has agreed to donate AWS and/or GCP comput=
e
> >>> and storage resources as well as manage the infrastructure.  These
> >>> project resources would be available for use by all committers and
> >>> community members regardless of corporate affiliations.  Note that
> >>> AFAIK there is no explicit requirement to host CI on ASF
> >>> infrastructure=E2=80=94unlike for critical project resources such as =
source
> >>> code, mailing lists, and issue tracking.
> >>>
> >>> The source for the pipeline and job scripts would reside within the
> >>> geode-* repos.  Geode committers would be able to modify those, same
> >>> as with our .travis.yml scripts.  All test results and build artifact=
s
> >>> would be publicly viewable just like with our Jenkins build output
> >>> today.  Requests for admin assistance would go through the dev@geode
> >>> mailing list.
> >>>
> >>> Thoughts?  As a first step we could run both CI systems side-by-side
> >>> and see how the Concourse approach works for our project.
> >>>
> >>> Thanks,
> >>> Anthony
> >>>
> >>>
> >>> [1] https://builds.apache.org/job/Geode-nightly/
> >>> [2] https://concourse.ci
> >>> [3] https://ci.concourse.ci
> >>> [4] https://bosh.io
> >>>
> >>
> >
> >
> >
> > --
> > Kindest Regards
> > -----------------------------
> > *Udo Kohlmeyer* | *Snr Solutions Architect* |*Pivotal*
> > *Mobile:* +61 409-279-160 | ukohlmeyer@pivotal.io
> > <http://www.gopivotal.com/>
> > www.pivotal.io
>
>

--94eb2c0724429ee009055ae554c2--