jackrabbit-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tobias Bocanegra" <tobias.bocane...@day.com>
Subject Fwd: Outage on www.apache.org, svn.apache.org, and other related infrastructure
Date Fri, 12 May 2006 12:35:01 GMT
FYI

---------- Forwarded message ----------
From: Garrett Rooney <rooneg@apache.org>
Date: May 11, 2006 7:08 AM
Subject: Outage on www.apache.org, svn.apache.org, and other related
infrastructure
To: committers@apache.org
Cc: infrastructure@apache.org


We've been having a heck of a day in infrastructure land, as you may
have noticed if you tried to access the Subversion repository or any
of the web sites...

Early this morning (PDT) minotaur, the machine that hosts
svn.apache.org, the ASF web sites, people.apache.org, and various
other things, kernel paniced.  After it was brought back up there was
some odd behavior observed, random programs aborting, stuff like that.

As a result, we decided to take some action before bringing all
services back online.  The data stored on the machine has been backed
up, both to ajax (the european backup server) and to helios, another
machine in the same datacenter as minotaur.  We've also run 'svnadmin
verify' on the repositories, both on the backup copies and on minotaur
itself, to confirm that whatever is wrong with minotaur has had no ill
effects.  We also ran memtest86 tests on minotaur, and have found no
sign of memory failure, which was our primary fear.

During this time, DNS for the websites served off of minotaur was
failed over to ajax, but svn.apache.org, perl.apache.org, and
tcl.apache.org remained down.  Subversion stayed down because we
needed to verify that the repository was ok before doing anything with
it, and TCL and Perl remained down because they require special setups
that are not mirrored on ajax.

At this point we have brought minotaur back to multiuser mode, and the
mail that was backed up on it (@apache.org mailing lists and
addresses) is flowing again.  The Subversion repositories have been
verified to be ok, there was no corruption found.  Soon the Subversion
repository will be brought back up in a read-only mode.  We are also
planning on upgrading minotaur to a newer version of FreeBSD, because
our best guess for what caused the initial problem is now a kernel
bug.  Once that is done we will likely turn write access to the
Subversion repository back on.  Sometime in the near future the
Subversion repository will be moved to an entirely new machine, but
the details of when and how that will happen are still being
determined.

Thank you for your patience during this outage,

The Apache Infrastructure Team


-- 
-----------------------------------------< tobias.bocanegra@day.com >---
Tobias Bocanegra, Day Management AG, Barfuesserplatz 6, CH - 4001 Basel
T +41 61 226 98 98, F +41 61 226 98 97
-----------------------------------------------< http://www.day.com >---

Mime
View raw message