www-builds mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Bayer <andrew.ba...@gmail.com>
Subject [COMPLETE] builds.apache.org maintenance on July 2nd
Date Sat, 04 Jul 2015 07:30:49 GMT
Hi all -

Finally! builds.apache.org is all the way back up and running. It's feeling
at least a bit peppier than it did before, and startup time was
significantly faster. Sorry again for the inconvenience of the extended
downtime - getting off that problematic ZFS volume that made the backup so
slow is exactly why we wanted to do this! I'd expect to see a little
weirdness with most recent builds in some jobs, but nothing that shouldn't
clear up after a new build runs. If you see anything more strange, please
let me know and I'll investigate. Thanks again!


On Fri, Jul 3, 2015 at 6:29 PM, Andrew Bayer <andrew.bayer@gmail.com> wrote:

> Hi all -
> Regrettably, the restore from backup is taking dramatically longer than
> expected - we're now not even halfway through, so I don't expect to be back
> online until Saturday morning UTC.
> A.
> On Fri, Jul 3, 2015 at 9:35 AM, Andrew Bayer <andrew.bayer@gmail.com>
> wrote:
>> Hi all -
>> Sorry for the delay in updating - we were finally able to get the OS
>> installed and booting properly, and are now in the process of restoring
>> Jenkins from backup. This'll take a few hours at least - sorry for the
>> inconvenience, but we should be back up this evening Europe time at the
>> outside.
>> A.
>> On Thu, Jul 2, 2015 at 5:17 PM, Andrew Bayer <andrew.bayer@gmail.com>
>> wrote:
>>> Hi all -
>>> builds.apache.org is still down - we're having problems rebuilding it.
>>> We don't yet have a timeframe for when it'll be back up, but once it's been
>>> reinstalled, I'll send another update. Thanks.
>>> A.
>>> On Thu, Jul 2, 2015 at 1:28 PM, Andrew Bayer <andrew.bayer@gmail.com>
>>> wrote:
>>>> Hooray - a correction. =) I've been able to get all the projects to
>>>> backup in a reasonable timeframe, so we should hopefully not be missing
>>>> builds. Sorry for the false alarm!
>>>> A.
>>>> On Thu, Jul 2, 2015 at 12:37 PM, Andrew Bayer <andrew.bayer@gmail.com>
>>>> wrote:
>>>>> builds.apache.org has been down for the last two hours as we work on
>>>>> finalizing the backup - due to the absolutely atrocious performance we've
>>>>> seen on this box with ZFS on Linux and directory traversal, we may not
>>>>> able to back up Maven project type builds from around 5pm UTC yesterday
>>>>> until Jenkins was brought offline at around 8:30am UTC today. To get
all of
>>>>> those backed up, it might take another 4 hours of rsyncing. The same
>>>>> numbers won't be repeated - just that any builds run in that time may
>>>>> be preserved. This will only be the case for Maven project type builds,
>>>>> freestyle jobs, and all previous builds and job configuration etc will
>>>>> still be there regardless. More updates when we're further along.
>>>>> A.
>>>>> On Tue, Jun 30, 2015 at 11:52 AM, Andrew Bayer <abayer@apache.org>
>>>>> wrote:
>>>>>> A reminder - builds.apache.org will be down on July 2nd, starting
>>>>>> around 9am UTC/11am CEST/10am BST/5am EDT/2am PDT. We'll be taking
>>>>>> Jenkins, backing it up, rebuilding the machine, and restoring Jenkins.
>>>>>> Hopefully this will be done within 5-6 hours, but could take longer.
>>>>>> send updates on the status as we go. Thanks.
>>>>>> If you have any questions please contact infrastructure@apache.org.
>>>>>> A.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message