www-builds mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Bayer <andrew.ba...@gmail.com>
Subject Re: [NOTICE] builds.apache.org maintenance on July 2nd
Date Sun, 05 Jul 2015 08:41:32 GMT
I also boosted the number of cloud slaves at a time, though I need to work
on that further. =)

A.

On Sunday, July 5, 2015, Hervé BOUTEMY <herve.boutemy@free.fr> wrote:

> great work!
> the build queue seems smaller now
>
> thank you Andrew
>
> Le samedi 4 juillet 2015 09:30:49 Andrew Bayer a écrit :
> > Hi all -
> >
> > Finally! builds.apache.org is all the way back up and running. It's
> feeling
> > at least a bit peppier than it did before, and startup time was
> > significantly faster. Sorry again for the inconvenience of the extended
> > downtime - getting off that problematic ZFS volume that made the backup
> so
> > slow is exactly why we wanted to do this! I'd expect to see a little
> > weirdness with most recent builds in some jobs, but nothing that
> shouldn't
> > clear up after a new build runs. If you see anything more strange, please
> > let me know and I'll investigate. Thanks again!
> >
> > A.
> >
> > On Fri, Jul 3, 2015 at 6:29 PM, Andrew Bayer <andrew.bayer@gmail.com
> <javascript:;>> wrote:
> > > Hi all -
> > >
> > > Regrettably, the restore from backup is taking dramatically longer than
> > > expected - we're now not even halfway through, so I don't expect to be
> > > back
> > > online until Saturday morning UTC.
> > >
> > > A.
> > >
> > > On Fri, Jul 3, 2015 at 9:35 AM, Andrew Bayer <andrew.bayer@gmail.com
> <javascript:;>>
> > >
> > > wrote:
> > >> Hi all -
> > >>
> > >> Sorry for the delay in updating - we were finally able to get the OS
> > >> installed and booting properly, and are now in the process of
> restoring
> > >> Jenkins from backup. This'll take a few hours at least - sorry for the
> > >> inconvenience, but we should be back up this evening Europe time at
> the
> > >> outside.
> > >>
> > >> A.
> > >>
> > >> On Thu, Jul 2, 2015 at 5:17 PM, Andrew Bayer <andrew.bayer@gmail.com
> <javascript:;>>
> > >>
> > >> wrote:
> > >>> Hi all -
> > >>>
> > >>> builds.apache.org is still down - we're having problems rebuilding
> it.
> > >>> We don't yet have a timeframe for when it'll be back up, but once
> it's
> > >>> been
> > >>> reinstalled, I'll send another update. Thanks.
> > >>>
> > >>> A.
> > >>>
> > >>>
> > >>> On Thu, Jul 2, 2015 at 1:28 PM, Andrew Bayer <andrew.bayer@gmail.com
> <javascript:;>>
> > >>>
> > >>> wrote:
> > >>>> Hooray - a correction. =) I've been able to get all the projects
to
> > >>>> backup in a reasonable timeframe, so we should hopefully not be
> missing
> > >>>> builds. Sorry for the false alarm!
> > >>>>
> > >>>> A.
> > >>>>
> > >>>> On Thu, Jul 2, 2015 at 12:37 PM, Andrew Bayer <
> andrew.bayer@gmail.com <javascript:;>>
> > >>>>
> > >>>> wrote:
> > >>>>> builds.apache.org has been down for the last two hours as we
work
> on
> > >>>>> finalizing the backup - due to the absolutely atrocious performance
> > >>>>> we've
> > >>>>> seen on this box with ZFS on Linux and directory traversal,
we may
> not
> > >>>>> be
> > >>>>> able to back up Maven project type builds from around 5pm UTC
> > >>>>> yesterday
> > >>>>> until Jenkins was brought offline at around 8:30am UTC today.
To
> get
> > >>>>> all of
> > >>>>> those backed up, it might take another 4 hours of rsyncing.
The
> same
> > >>>>> build
> > >>>>> numbers won't be repeated - just that any builds run in that
time
> may
> > >>>>> not
> > >>>>> be preserved. This will only be the case for Maven project
type
> > >>>>> builds, not
> > >>>>> freestyle jobs, and all previous builds and job configuration
etc
> will
> > >>>>> still be there regardless. More updates when we're further
along.
> > >>>>>
> > >>>>> A.
> > >>>>>
> > >>>>> On Tue, Jun 30, 2015 at 11:52 AM, Andrew Bayer <abayer@apache.org
> <javascript:;>>
> > >>>>>
> > >>>>> wrote:
> > >>>>>> A reminder - builds.apache.org will be down on July 2nd,
starting
> > >>>>>> around 9am UTC/11am CEST/10am BST/5am EDT/2am PDT. We'll
be taking
> > >>>>>> down
> > >>>>>> Jenkins, backing it up, rebuilding the machine, and restoring
> > >>>>>> Jenkins.
> > >>>>>> Hopefully this will be done within 5-6 hours, but could
take
> longer.
> > >>>>>> We'll
> > >>>>>> send updates on the status as we go. Thanks.
> > >>>>>>
> > >>>>>> If you have any questions please contact
> infrastructure@apache.org <javascript:;>.
> > >>>>>>
> > >>>>>> A.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message