spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reynold Xin <>
Subject Re: [build system] massive jenkins infrastructure changes forthcoming
Date Fri, 18 Nov 2016 00:52:22 GMT
Thanks for the headsup, Shane.

On Thu, Nov 17, 2016 at 2:33 PM, shane knapp <> wrote:

> TL;DR:  amplab is becomine riselab, and is much more C++ oriented.
> centos 6 is so far behind, and i'm already having to roll C++
> compilers and various libraries by hand.  centos 7 is an absolute
> no-go, so we'll be moving the jenkins workers over to a recent (TBD)
> version of ubuntu server.  also, we'll finally get jenkins upgraded to
> the latest LTS version, as well as our insanely out of date plugins.
> riselab (me) will still run the build system, btw.
> oh, we'll also have a macOS worker!
> well, that was still pretty long.  :)
> anyways, you have the gist of it.  this is something we're going to do
> slowly, so as to not interrupt any spark, alluxio or lab builds.
> i'll be spinning up a master and two worker ubuntu nodes, and then
> port a couple of builds over and get the major kinks worked out.
> then, early next year, we can point the new master at the old workers,
> and one-by-one reinstall and deploy them w/ubuntu.
> i'll be reaching out to some individuals (you know who you are) as
> things progress.
> if we do this right, we'll have minimal service interruptions and end
> up w/a clean and fresh jenkins.  this is the opposite of our current
> jenkins, which is at least 4 years old and is super-glued and
> duct-taped together.
> the ubuntu staging servers should be ready early next week, but i
> don't foresee much work happening until after thanksgiving.

View raw message