Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 15345200B27 for ; Wed, 22 Jun 2016 16:25:47 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 13A0A160A36; Wed, 22 Jun 2016 14:25:47 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id D89EA160A24 for ; Wed, 22 Jun 2016 16:25:45 +0200 (CEST) Received: (qmail 74842 invoked by uid 500); 22 Jun 2016 14:25:40 -0000 Mailing-List: contact users-help@subversion.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list users@subversion.apache.org Received: (qmail 74829 invoked by uid 99); 22 Jun 2016 14:25:39 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2016 14:25:39 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 53826C17C6 for ; Wed, 22 Jun 2016 14:25:39 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.198 X-Spam-Level: * X-Spam-Status: No, score=1.198 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id spyFqnvkBEcr for ; Wed, 22 Jun 2016 14:25:37 +0000 (UTC) Received: from mail-vk0-f47.google.com (mail-vk0-f47.google.com [209.85.213.47]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with ESMTPS id A3A6C5F3FC for ; Wed, 22 Jun 2016 14:25:36 +0000 (UTC) Received: by mail-vk0-f47.google.com with SMTP id d185so64101076vkg.0 for ; Wed, 22 Jun 2016 07:25:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=I4CBI9TO9BcP4bkAYVwNV3LpBP33FUFlxzYFKJimtDY=; b=rDXjIXKgfcEBF0G3ZK0m13Er+O5U7Elx+/i2MpzFsIB2ngcZF66p/WICR1ut9G6KmY 8BfrRdBvKb0rNN0AePPfN1zW2/Krx9GZV0J9r9hy+U87IlC+QEX92erfISdhM63zo/x9 HhRdQzZoXhkXmZ3s0+8w9P857MDXwEamMGesB8voMZbNhzbG6cw0HOrlm4TCl5nZggVd vTnRrC2NGLE7ABZhSxnMInN+T6xSmscK3GMiaugJC0Xf6ryv2DCQ+lXsFPx5j5hBrk3j OmoFPoMrXzp2glIEZeDWNAUnUbQxtMJkpxlwR/kM8BUEqQyd3zY9V2y7CzRLXKgtABEN EedQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=I4CBI9TO9BcP4bkAYVwNV3LpBP33FUFlxzYFKJimtDY=; b=GXGfeZBohgWOBT4+eYy2yE+mq3fBSyZEbiP/+67lWz0ap28zRrOOfCuvcB8dSwXAyw /TrsNy3Npe1aG8DmXERf2CfeLhNvMFf5q0rA1ZBLR+Cc43KVxjfqiuas6wqf5OQ5hHq4 5Ad7Zrv0ieSw8ttsYwNCniP9cDNCIY8+1nddYCOATHxkW6fqnbvL9TJiXrgEvTTkL3ez HBx0zQczFCHWR3BB0hmOTYANDKMTIR1jDGORijbw84gjZMnFRZi/6Q+ocbSeX+MYiFmD ZS3SPUUFJlJQBoX8Sx7XKeH1q0fb/jSmU2W3DoLknCth7pO551NZ2n1thoe1Jjc3Ie+w 5TEQ== X-Gm-Message-State: ALyK8tI19boPfCvb3jnklW0yHyN8/SDytAsrkLyK6HZef9GLl4f/NdVKlkHGnqiDFln5v8HQMLljDdk9Xf796Q== X-Received: by 10.31.10.82 with SMTP id 79mr10322494vkk.67.1466605529340; Wed, 22 Jun 2016 07:25:29 -0700 (PDT) MIME-Version: 1.0 Received: by 10.176.0.72 with HTTP; Wed, 22 Jun 2016 07:25:28 -0700 (PDT) In-Reply-To: References: From: Michael Schwager Date: Wed, 22 Jun 2016 09:25:28 -0500 Message-ID: Subject: Re: Creating and Verifying a Reliable backup To: users@subversion.apache.org Content-Type: multipart/alternative; boundary=001a11457d649a23750535deb57c archived-at: Wed, 22 Jun 2016 14:25:47 -0000 --001a11457d649a23750535deb57c Content-Type: text/plain; charset=UTF-8 Following is an update to my question of Jun 1, where I ask the following question: On Wed, Jun 1, 2016 at 9:58 AM, Michael Schwager wrote: > ... > My question is: How do I back (subversion repos) up reliably, and verify > (them) so that I can deliver a 100% recovery guarantee to my boss? > ... > I have compiled subversion-1.9.4 on the server under > /opt/subversion-1.9.4. If I run that version of svn hotcopy, it appears to > work and svnverify exits successfully. But ... I find that a file is > missing: repos2/db/rev-prop-atomics.shm . > I have implemented a script that performs the following, in abbreviated Bash form. Note that I have enough disk space for the original repos directories, plus 2 copies. I am backup up the whole enchilada to tape. And archiving it to another pair of tapes monthly and taking one of them offsite. There... that oughta do it :-) . Without further ado: LOGFILE=/var/log/backup_svn.$(date +%a) exec 1>&$LOGFILE exec 2>&1 REPODIR=/home/svn/svn/repositories HOTCOPYDIR=/home/svn/hotcopy RSYNCDIR=/home/svn/rsync RELIABLE_SVNADMIN=/opt/subversion-1.9.4/bin/svnadmin EMAIL=username@example.com # $repos is a space-separated list of all the directories in /home/svn/svn/repositories for repo in $repos; do # # VERIFY ORIGINAL ---------------------------------------------------------------------- # echo "*********************************************************** *********************" echo "**** REPO $repo pre-backup sanity check ***************************************" echo "********************************************************************************" svnadmin verify $repo || { mail -s "ERROR: SVN directory FATAL error. This is a BIG problem. Backups are HALTED." $EMAIL < $LOGFILE; exit 1; } # # HOTCOPY ------------------------------------------------------------------------------ # echo "********************************************************************************" echo "**** REPO $repo hotcopy *******************************************************" echo "********************************************************************************" rm -rf $HOTCOPYDIR/$repo.1 [[ -e $HOTCOPYDIR/$repo ]] && mv $HOTCOPYDIR/$repo $HOTCOPYDIR/$repo.1 $RELIABLE_SVNADMIN hotcopy $REPODIR/$repo $HOTCOPYDIR/$repo || { mail -s "ERROR: svn hotcopy exited with nonzero status." $EMAIL < $LOGFILE; exit 1; } echo "********************************************************************************" echo "**** REPO $repo hotcopy verify ************************************************" echo "********************************************************************************" svnadmin verify $HOTCOPYDIR/$repo || { mail -s "ERROR: SVN hotcopy error. The original is fine, hotcopy is not." $EMAIL < $LOGFILE; exit 1; } # # RSYNC -------------------------------------------------------------------------------- # echo "********************************************************************************" echo "**** REPO $repo rsync *********************************************************" echo "********************************************************************************" rm -rf $RSYNCDIR/$repo.1 [[ -e $RSYNCDIR/$repo ]] && mv $RSYNCDIR/$repo $RSYNCDIR/$repo.1 $RELIABLE_SVNADMIN freeze $repo -- rsync -av $repo $RSYNCDIR/ || { mail -s "ERROR: rsync of svn repo $repo exited with nonzero status." $EMAIL < $LOGFILE; exit 1; } echo "********************************************************************************" echo "**** REPO $repo rsync verify **************************************************" echo "********************************************************************************" svnadmin verify $RSYNCDIR/$repo || { mail -s "ERROR: SVN rsync error. The original is fine, rsync is not." $EMAIL < $LOGFILE; exit 1; } echo "********************************************************************************" echo "**** Do other directories: authorization and bin *******************************" echo "********************************************************************************" cd $REPODIR/.. rm -rf $RSYNCDIR/authorization.1 [[ -e $RSYNCDIR/authorization ]] && mv $RSYNCDIR/authorization $RSYNCDIR/authorization.1 rm -rf $RSYNCDIR/bin.1 [[ -e $RSYNCDIR/bin ]] && mv $RSYNCDIR/bin $RSYNCDIR/bin.1 rsync -av authorization $RSYNCDIR/ rsync -av bin $RSYNCDIR/ -- -Mike Schwager --001a11457d649a23750535deb57c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Foll= owing is an update to my question of Jun 1, where I ask the following quest= ion:

O= n Wed, Jun 1, 2016 at 9:58 AM, Michael Schwager <mschwage@gmail.com&g= t; wrote:
...
My question i= s: How do I back (subversion repos) up reliably, and verify (them) so that = I can deliver a 100% recovery guarantee to my boss?
...
I have c= ompiled subversion-1.9.4 on the server under /opt/subversion-1.9.4. If I ru= n that version of svn hotcopy, it appears to work and svnverify exits succe= ssfully. But ... I find that a file is missing:=C2=A0repos2/db/rev-prop-ato= mics.shm .

I have implemented a script that performs the following, i= n abbreviated Bash form. Note that I have enough disk space for the origina= l repos directories, plus 2 copies. I am backup up the whole enchilada to t= ape. And archiving it to another pair of tapes monthly and taking one of th= em offsite. There... that oughta do it :-) . Without further ado:

LOGFILE=3D/var/log/backup_svn.$(date +%a)
exec 1>&$LOGFILE
exec 2>&1
REPODIR=3D/home/svn/svn/repositories
HOTCOPYDIR=3D/home/s= vn/hotcopy
RSYNCDIR=3D/home/svn/rsync
RELIABLE_SVNADMIN=3D/opt/subversion-1= .9.4/bin/svnadmin
EMAIL=3Dusern= ame@example.com
# $repos is a space-separated list of all the directori= es in /home/svn/svn/repositories
for repo in $repos; do
=C2=A0 =C2=A0 =C2=A0 =C2=A0 #
=C2=A0 =C2=A0 =C2=A0 =C2=A0 # VER= IFY ORIGINAL --------------------------------------------------------------= --------
=C2=A0 =C2=A0 =C2=A0 =C2=A0 #
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 echo "********************************************************************************"
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 echo "**** REPO $repo pre-backup sanity check ***= ************************************"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 echo= "********************************************************************= ************"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 svnadmin verify $repo || { m= ail -s "ERROR: SVN directory FATAL error. This is a BIG problem. Backu= ps are HALTED." $EMAIL < $LOGFILE;=C2=A0exit 1; }

=C2= =A0 =C2=A0 =C2=A0 =C2=A0 #
=C2=A0 =C2=A0 =C2=A0 =C2=A0 # HOTCOPY ----------= --------------------------------------------------------------------=
=C2=A0= =C2=A0 =C2=A0 =C2=A0 #
=C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "**********= **********************************************************************"= ;
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 echo "*********************************************************= ***********************"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 rm -rf $HOTCOPYDI= R/$repo.1
=C2=A0 =C2=A0 =C2=A0 =C2=A0 [[ -e $HOTCOPYDIR/$repo ]] &&= mv $HOTCOPYDIR/$repo $HOTCOPYDIR/$repo.1
=C2=A0 =C2=A0 =C2=A0 =C2=A0 $RELI= ABLE_SVNADMIN hotcopy $REPODIR/$repo $HOTCOPYDIR/$repo || { mail -s "E= RROR: svn hotcopy exited with nonzero status." $EMAIL < $LOGFILE; exit 1; }
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 echo "**********************************************= **********************************"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 echo &= quot;**** REPO $repo hotcopy verify ***************************************= *********"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "********************= ************************************************************"
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 svnadmin verify $HOTCOPYDIR/$repo || { mail -s "E= RROR: SVN hotcopy error. The original is fine, hotcopy is not." $EMAIL= < $LOGFILE;=C2=A0exit 1; }

=C2=A0 =C2=A0 =C2=A0 =C2=A0 #
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 # RSYNC ---------------------------------------= -----------------------------------------
=C2=A0 =C2=A0 =C2=A0 =C2=A0 #
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 echo "***************************************= *****************************************"
=C2=A0 =C2=A0 =C2=A0 =C2=A0= echo "**** REPO $repo rsync *****************************************= ****************"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "*************= *******************************************************************"
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 rm -rf $RSYNCDIR/$repo.1
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 [[ -e $RSYNCDIR/$repo ]] && mv $RSYNCDIR/$repo =C2=A0$RSYNCD= IR/$repo.1
=C2=A0 =C2=A0 =C2=A0 =C2=A0 $RELIABLE_SVNADMIN freeze $repo -- r= sync -av $repo $RSYNCDIR/ || { mail -s "ERROR: rsync of svn repo $repo= exited with nonzero status." $EMAIL < $LOGFILE; exit 1; }
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = echo "****************************************************************= ****************"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "**** REPO $re= po rsync verify **************************************************"
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "************************************= ********************************************"
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 svnadmin verify $RSYNCDIR/$repo || { mail -s "ERROR: SVN rsync = error. The original is fine, rsync is not." $EMAIL < $LOGFILE; exit= 1; }

=C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "******************= **************************************************************"=
=C2=A0= =C2=A0 =C2=A0 =C2=A0 echo "**** Do other directories: authorization a= nd bin *******************************"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 ec= ho "******************************************************************= **************"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 cd $REPODIR/..
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 rm -rf $RSYNCDIR/authorization.1
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 [[ -e $RSYNCDIR/authorization ]] && mv $RSYNCDIR/authorizati= on $RSYNCDIR/authorization.1
=C2=A0 =C2=A0 =C2=A0 =C2=A0 rm -rf $RSYNCDIR/b= in.1
=C2=A0 =C2=A0 =C2=A0 =C2=A0 [[ -e $RSYNCDIR/bin ]] && mv $RSYN= CDIR/bin $RSYNCDIR/bin.1
=C2=A0 =C2=A0 =C2=A0 =C2=A0 rsync -av authorizati= on $RSYNCDIR/
=C2=A0 =C2=A0 =C2=A0 =C2=A0 rsync -av bin $RSYNCDIR/


<= /div>--
-Mike Schwager
--001a11457d649a23750535deb57c--