zipkin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zoltán Nagy <abes...@gmail.com>
Subject Re: Infrastructure setup / CI
Date Thu, 06 Sep 2018 07:36:27 GMT
Thanks for sharing your thoughts, Andrey!

I'm still looking to hear opinions from other Zipkin committers. In the
meanwhile, my two cents.

Re. moving the GitHub repositories to under the Apache org, this should be
fine IMHO. But let's wait to hear from Adrian on this. Meta: how are
questions like this decided under the ASF? Is there a standard decision
procedure here, or is it up to each project to formalize how decisions like
this are made? Is there even need for formalizing the process?

At a glance, it looks like ASF Jenkins is running a recent version of
Jenkins, including pipelines (both declarative and the other one whose name
I always forget) and multibranch pipelines, with static build nodes
(documented at
https://cwiki.apache.org/confluence/display/INFRA/Jenkins+node+labels).
There's no dynamic (Docker-based or otherwise) provisioning of executors,
meaning the build environment is what it is, with little room for
per-project customization. I see 24 nodes running Xenial, with 2 executors
per node, which is what we'd use, probably exclusively. I didn't find
analytics on queue times, but looking at the build queue for a couple of
minutes, it seems to me that queue times are on the minute scale. Is that a
fair summary?

As for the other options listed on https://ci.apache.org/ (namely Buildbot
and Gump), I feel we'd be best served by Jenkins of these three; OpenZipkin
is JVM-heavy, so Jenkins is a “native” CI system. I'm slightly confused
about why https://ci.apache.org/ doesn't list Travis CI.

My personal stance at this point: infrastructure-wise, the two things we'd
need to evaluate for feasibility is queue times and environment
customization (are all our build-time system dependencies available? if
not, can we replace them with present alternatives, or get them into the
executors?). I'd probably be doing the majority of the migration work, at
least in the PoC and initial migration phase, so I want to note that I'm OK
with that amount of work. At the same time, this would be a big change in
the way OpenZipkin development is done (Travis to Jenkins), and I'm eager
to hear from other committers how you all feel about the mental overhead
(and unavoidable early migration-related breakages) of a migration like
this. There's value to it, but it's significantly more expensive than
moving everything to Travis.

Cheers!

On Tue, Sep 4, 2018 at 5:31 PM Andrey Redko <drreta@gmail.com> wrote:

> Hey Zoltán,
>
> Indeed, you have brought a lot of great points to discuss!
>
> I hope other guys will correct me if I am wrong, but as per
> https://incubator.apache.org/guides/podling_sourcecontrol.html, the
> repositories should be hosted by Apache or on Github under Apache
> organization. The latter should be fine, right?
>
> Regarding CI, most project use Jenkins (builds.apache.org), the
> ci.apache.org is the source for CI infra for ASF. Would it work for you
> guys (assuming the help with configuring Jenkins will be surely offered).
>
> What do you think?
>
> Best Regards,
>     Andriy Redko
>
> On Tue, Sep 4, 2018, 4:35 AM Zoltán Nagy <abesto0@gmail.com> wrote:
>
> > Hey folks,
> >
> > So great to send a mail to a list whose address includes both Zipkin and
> > Apache :)
> > Watch out, wall of text incoming. There's a lot of context here.
> >
> > After reading the previous mail threads (still before the list), and some
> > discussion with Andrew (whom Adrian introduced, and was a great help, are
> > you here, Andrew?), I'm starting to have an outline of infrastructure-y
> > things that need to happen.
> >
> > Most of these are pretty straightforward, and I'll try to capture
> existing
> > stated preferences – do shout if I'm telling a lie.
> >
> >    - *Git repository hosting*. OpenZipkin people AFAIK would prefer
> staying
> >    on GitHub if at all possible. This would imply setting up syncing to
> ASF
> >    git hosting. https://gitbox.apache.org/ seems to be a solution to
> this
> >    whole can of worms, though I didn't look too deeply yet.
> >    - Release automation may need to be extended for the required *voting
> >    process* – or maybe just leave it up to human process, don't change
> the
> >    automation.
> >    - We'll need to set up *automated notification e-mails* about commits,
> >    releases, that kind of stuff. For commits, it should come out of the
> box
> >    with the Git setup, and should be easy to add to the release
> automation.
> >    This is probably best done with a separate notifications@ list.
> >
> > Now for the hard part: *CI*. Travis seems to be supported by ASF, but
> > CircleCI is not, and various OpenZipkin projects currently use one, the
> > other, or even both. ASF apparently also has a hosted Jenkins server
> (which
> > I can't find ATM).
> >
> > *A question to mentors*: what other CI options are available under the
> ASF
> > umbrella? Links to the various solutions or documentation for them would
> be
> > especially appreciated, so we can evaluate them properly.
> >
> > *A question to OpenZipkin folks*: what's your current take on our CI
> setup?
> > Here too, let me share the perspectives I've heard so far, call out any
> > mistakes I make:
> >
> >    - Travis sometimes has stability problems (more often than “rarely”,
> but
> >    more rarely than “often”, and more often than Circle)
> >    - Circle is somewhat more featureful and stable, but is severely
> limited
> >    in the number of free executors (and again, is not currently
> > ASF-approved)
> >    - The less we need to care about CI, the better – that includes
> ongoing
> >    maintenance, mental / process overhead when making changes, and cost
> of
> > any
> >    migration
> >    - This one's mostly from me: there's a lot of duplication between the
> >    build definitions of our various projects, not to mention the overhead
> > of
> >    “supporting” both Travis and Circle. For example, I know Jenkins has
> >    standard tools to deduplicate parts of build definitions (namely,
> shared
> >    libraries for pipelines); I don't know of similar solutions for Travis
> > (and
> >    am happy to be pointed at some / do more research).
> >
> > I'd say using a single CI system is more of a “when”, rather than a
> > “whether” type of question. It then boils down to a trade-off between the
> > cost of migration, and the perceived value of the new system. Some
> possible
> > scenarios:
> >
> >    - We decide to minimize the cost of migration, and move Circle builds
> to
> >    Travis. Later on, we might look into deduplicating build logic between
> > jobs.
> >    - We push for ASF to accept CircleCI (note, with no guarantee of
> success
> >    at this point), including funding for more executors if needed (or try
> > to
> >    start a collaboration with Circle so that ASF projects get more
> > executors),
> >    and migrate Travis jobs to Circle.
> >    - We decide to maximize perceived value, by choosing a third option
> >    (non-Travis, non-Circle) that helps DRY our build definitions, while
> > also
> >    keeping it easy to make per-project changes as needed. I'd recommend
> >    leading a project like this with a PoC and evaluating the results
> there,
> >    before migrating all N>10 projects and pouring effort into finalizing
> > the
> >    shared abstraction layer.
> >
> > EOF wall of text. To recap, I'm super interested in what other CI options
> > are available under ASF (so that we know what to even think about), any
> > other recommendations our beloved mentors might have given all this, and
> > preferences / arguments / corrections from Zipkin folks.
> >
> > Cheers!
> >
> > --
> > Zoltán Nagy
> > https://abesto.net
> >
>


-- 
Zoltán Nagy
https://abesto.net

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message