www-builds mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer ...@effectivemachines.com>
Subject Re: Turning Off Bb-fbsd2
Date Fri, 10 Feb 2017 01:38:17 GMT

> On Feb 9, 2017, at 4:32 PM, Greg Stein <gstein@gmail.com> wrote:
> I also believe we had to pay for that box, and it wasn't cheap.

	I had it in my head that it was from the Apple hardware donation that happened a while back,
but that was a long time ago.  (So long ago that it appears to have been wiped from the ASF

> Today, our preferred model for non-Ubuntu boxes is to have other people
> own/run/manage those buildbots and hook them into our build master.

	Yup. That's how the PowerPC got added. Thus why I say it's better for projects to start digging
into corporate sponsorship. FWIW: It was not a smooth process, but there were issues on both
the donate-ees and the donate-ers.

> Apache Hadoop is worse for it. As Gavin has noted, just in the past year,
> we've increased our build farm dramatically. I believe the ASF is better
> for it.

	I guess I'm viewing this from a different and very selfish perspective. Prior to your time,
a lot of effort was spent by the Hadoop project on reducing the build infrastructure footprint
because the project itself was getting bogged down, build failures (esp after 2 executors
magically appeared on the nodes), etc, etc. We moved all/most of our builds to be docker-ized
to also minimize the impact on the infra team.  End result was significant savings all around.
 Many jobs were shutdown, slots were opened up, builds became more reliable, Yetus was born
to share our experiences, etc, etc.  The ecosystem of projects that were running on those
nodes also benefitted since Hadoop was usually the #1 or #2 user.   That said: last I checked,
there have been no new build machines added to the Hadoop pool.  In fact, we usually run at
a deficit because machines are always down. (right now, we're at something like 60% capacity:
H2,H3,H7,H8,H9 are all offline).  So while the build farm may have increased for the ASF overall,
Hadoop is not benefitting from a hardware increase.  We've benefitted from fixing our own
build jobs and from the infra team's work on Jenkins itself.

>  Today, if a machine goes down,
> we can spin it back up in an hour or two due to the consistency.

	Great, so I should expect for all of our nodes to come back up quickly then, right? ;)

> I do sympathize that our service reduction is painful. But I hope you can
> understand where the Foundation (and its Infra team) is coming from. We
> have vastly more projects to support today, meaning more uniformity is
> required.

	It is what it is.  I'm not angry (anymore). While I do really appreciate a lot of what the
infra team goes through (I'm much more of an ops person than a dev person), it just needs
to be pointed out that there are impacts to teams and that everything isn't completely 100%
positive. I've resigned myself to the fact that it's better for my own sanity to donate my
time to other things than the ASF.  Most of that is related to how the Apache Hadoop project
itself is run, but there is a portion of that related to getting burned by the infra team's
changes. I think things are improving, but I'm still very much in a wait-and-see mode.

View raw message