www-builds mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Bayer <andrew.ba...@gmail.com>
Subject [COMPLETE] builds.apache.org maintenance on July 2nd
Date Sat, 04 Jul 2015 07:30:49 GMT
Hi all -

Finally! builds.apache.org is all the way back up and running. It's feeling
at least a bit peppier than it did before, and startup time was
significantly faster. Sorry again for the inconvenience of the extended
downtime - getting off that problematic ZFS volume that made the backup so
slow is exactly why we wanted to do this! I'd expect to see a little
weirdness with most recent builds in some jobs, but nothing that shouldn't
clear up after a new build runs. If you see anything more strange, please
let me know and I'll investigate. Thanks again!

A.

On Fri, Jul 3, 2015 at 6:29 PM, Andrew Bayer <andrew.bayer@gmail.com> wrote:

> Hi all -
>
> Regrettably, the restore from backup is taking dramatically longer than
> expected - we're now not even halfway through, so I don't expect to be back
> online until Saturday morning UTC.
>
> A.
>
> On Fri, Jul 3, 2015 at 9:35 AM, Andrew Bayer <andrew.bayer@gmail.com>
> wrote:
>
>> Hi all -
>>
>> Sorry for the delay in updating - we were finally able to get the OS
>> installed and booting properly, and are now in the process of restoring
>> Jenkins from backup. This'll take a few hours at least - sorry for the
>> inconvenience, but we should be back up this evening Europe time at the
>> outside.
>>
>> A.
>>
>> On Thu, Jul 2, 2015 at 5:17 PM, Andrew Bayer <andrew.bayer@gmail.com>
>> wrote:
>>
>>> Hi all -
>>>
>>> builds.apache.org is still down - we're having problems rebuilding it.
>>> We don't yet have a timeframe for when it'll be back up, but once it's been
>>> reinstalled, I'll send another update. Thanks.
>>>
>>> A.
>>>
>>>
>>> On Thu, Jul 2, 2015 at 1:28 PM, Andrew Bayer <andrew.bayer@gmail.com>
>>> wrote:
>>>
>>>> Hooray - a correction. =) I've been able to get all the projects to
>>>> backup in a reasonable timeframe, so we should hopefully not be missing
>>>> builds. Sorry for the false alarm!
>>>>
>>>> A.
>>>>
>>>> On Thu, Jul 2, 2015 at 12:37 PM, Andrew Bayer <andrew.bayer@gmail.com>
>>>> wrote:
>>>>
>>>>> builds.apache.org has been down for the last two hours as we work on
>>>>> finalizing the backup - due to the absolutely atrocious performance we've
>>>>> seen on this box with ZFS on Linux and directory traversal, we may not
be
>>>>> able to back up Maven project type builds from around 5pm UTC yesterday
>>>>> until Jenkins was brought offline at around 8:30am UTC today. To get
all of
>>>>> those backed up, it might take another 4 hours of rsyncing. The same
build
>>>>> numbers won't be repeated - just that any builds run in that time may
not
>>>>> be preserved. This will only be the case for Maven project type builds,
not
>>>>> freestyle jobs, and all previous builds and job configuration etc will
>>>>> still be there regardless. More updates when we're further along.
>>>>>
>>>>> A.
>>>>>
>>>>> On Tue, Jun 30, 2015 at 11:52 AM, Andrew Bayer <abayer@apache.org>
>>>>> wrote:
>>>>>
>>>>>> A reminder - builds.apache.org will be down on July 2nd, starting
>>>>>> around 9am UTC/11am CEST/10am BST/5am EDT/2am PDT. We'll be taking
down
>>>>>> Jenkins, backing it up, rebuilding the machine, and restoring Jenkins.
>>>>>> Hopefully this will be done within 5-6 hours, but could take longer.
We'll
>>>>>> send updates on the status as we go. Thanks.
>>>>>>
>>>>>> If you have any questions please contact infrastructure@apache.org.
>>>>>>
>>>>>> A.
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message