mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Mahler <>
Subject Re: On the current CI state
Date Wed, 25 Oct 2017 00:38:32 GMT
Thanks Alex!

Also I would like to re-state the importance of everyone subscribing to the
builds@ list and helping triage the build failure emails. In particular, if
you find a ticket, reply with it so that others don't have to look into it.
If there's no ticket, capturing the logs of the bad run (and ideally also a
good run) and reply with the new ticket. This in itself is a big help!

On Mon, Oct 23, 2017 at 3:54 PM, Alex Rukletsov <> wrote:

> Folks,
> the CI state (both Apache and internal we have at Mesosphere) has recently
> degraded to a point when people no longer look at it failures. This defeats
> the primary purpose of a CI: to produce a reliable signal when a change
> breaks something.
> You might have seen a bunch of commits fixing flaky tests and bugs over the
> past two weeks — this is the beginning of our effort to bring the CI back
> to the green state. To track the effort, there exists a swim lane in our
> tech debt board [1] and a flow diagram [2]. I believe that some of the
> older tickets are no longer relevant, I will do a cleanup at some point
> when I get a better feeling of the actual state.
> If you would like to help, watch out for new flakiness new changes might
> introduce. Apache CI apparently has a quirk when a test run can pause for
> 15+s, leading to arbitrary test failures. This is a false positive, but the
> pattern is easily recognizabe in the logs.
> We also have a dedicated channel in Apache Mesos slack: #ci-back-to-green
> If you would like to participate, here is the list of the biggest offenders
> that are not triaged yet: MESOS-7519, MESOS-7082, MESOS-7434, MESOS-7512,
> MESOS-7742, MESOS-7028, MESOS-7425, MESOS-7106, MESOS-7337, MESOS-7273,
> MESOS-6724, MESOS-8112, MESOS-6949, MESOS-8000, MESOS-8047
> Alex.
> [1]
> rapidView=151&view=detail&selectedIssue=MESOS-8005
> [2]
> rapidView=204&view=reporting&chart=cumulativeFlowDiagram&
> swimlane=501&column=774&column=775&column=776&days=7

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message