www-mirrors mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karsten W. Rohrbach" <rohrb...@nacamar.net>
Subject rsync woes
Date Fri, 22 Sep 2000 18:29:08 GMT
hello fellow apache mirrors!

since i'm running a quite-not-so-tiny site in europe i want to share you
some of my rsync experiences and config stuff.

just a few points to consider:

-z does not really speed up the transfer, in the server config it should
be disabled generally
-rpltv should be the standard singelchar option string since it nearly
does everything one needs including linking and permission stuff, see
rsync(1) for explanation
--delete is mandatory
--safe-links is also mandatory for mirror sites (!!!)

to mirror apache-site one should exclude:
--exclude incoming/
--exclude bugs/

notice the trailing slash wich says, that these are directories

so for a plain vanilla mirror it boils down to
-----BEGIN FILE apache-site.rsync.sh-----
#!/bin/sh
# apache-site.rsync.sh -- rohrbach@nacamar.net -- 20000918
/usr/local/bin/rsync -rtvpl --safe-links --delete \
	--exclude incoming/ \
	--exclude bugs/ \
	rsync://dev.apache.org/apache-site/ \
	/home/service/ftp/mirrors/dev.apache.org
-----END FILE apache-site-rsync.sh-----

example output follows:
-----BEGIN EXAMPLE OUTPUT-----
mirror@WM:datasink[~/bin/24h]23% ./apache-site.rsync 

receiving file list ... done
wrote 121 bytes  read 32264 bytes  5888.18 bytes/sec
total size is 397241652  speedup is 12266.22
-----END EXAMPLE OUTPUT-----

on the server side one's got to look at a few things, especially
quenching client options for 
a) compression, since it means sysload
b) timeouts greater 5 minutes because of dos attacks
if your clients are fast, you can go below 30 secs with client timeouts,
i got a few russian peers which access my site as master, therefor it is 
5 minutes.

-----BEGIN FILE rsyncd.conf-----
# rsyncd.conf -- rohrbach@nacamar.net -- 20000320

# don't do any harm to the system
uid = nobody
gid = nogroup
use chroot = no
max connections = 40
pid file = /var/run/rsyncd.pid
motd file = /etc/rsyncd/motd
syslog facility = local7
read only = yes
timeout = 300

# quench clients' nasty options
dont compress = *
refuse options = timeout

# here go your [package] parameters...
[apache-ftp]
        path = /ftp/pub/unix/apache
        comment = [~0.3GB] Apache WWW Server FTP Archive
        list = yes

[apache-www]
        path = /meta/mirror/apache-www
        comment = [~0.4GB] Apache WWW Server WWW Site
        list = yes
#eof
-----END FILE rsyncd.conf-----

example output follows:
-----BEGIN EXAMPLE OUTPUT-----
mirror@WM:datasink[~/bin/24h]24% rsync rsync://filepile.nacamar.net
###
### You reached
### ____________.____  ______________________.____  ___________
### \_   _____/_|    | \_   _____/______   \_|    | \_   _____/
###  |    __)|  |    |  |    __)_ |     ___/ |    |  |    __)_
###  |     \ |  |    |__|        \|    |  |  |    |__|        \
###  \___  / |__|_______ \_____  /|____|  |__|_______ \_____  /
###      \/             \/     \/      .nacamar.net  \/     \/
###
###                        ____  ____ __________ _____ ___  ____ ______
###                       / __ \/ __ `/ ___/ __ `/ __ `__ \/ __ `/ ___/
###                      / / / / /_/ / /__/ /_/ / / / / / / /_/ / /
### one world. one net. /_/ /_/\__,_/\___/\__,_/_/ /_/ /_/\__,_/_/ .
[...]
package         [  size] description
=============== ======== ==============================================

apache-ftp     	[~0.3GB] Apache WWW Server FTP Archive
apache-www     	[~0.4GB] Apache WWW Server WWW Site
[...]
-----END EXAMPLE OUTPUT-----

i hope this answers some of the questions that came up in the last
months.

<shameless plug>

if you want to see a picture of the server carrying all rsync
im-/exports try 
	ftp://filepile.nacamar.net/info/
there you can see what i'm talking about, it has become quite
a big site blasting some gigs per day into the european countries:

Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda1              3842376    521684   3125504  14% /
/dev/sda2              1968620   1049652    918968  53% /swap
/dev/sda3             11820088   1383248   9836412  12% /home
/dev/sdb1             89591724   6150600  83441124   7% /meta
/dev/sdc1            125431656  48284732  77146924  38% /ftp/.vol/vol0
/dev/sdd1            125431656  49909256  75522400  40% /ftp/.vol/vol1
/dev/sde1            125431656  68579468  56852188  55% /ftp/.vol/vol2

</shameless plug>

if you are interested in my collection of mirroring scripts including
all the lock/rsync/ftpmirror and log management, just drop me a mail

have fun & keep up the good work!
/k

-- 
> Coders do it with a routine.
karsten w. rohrbach - research engineer - dept. strategies & development
nacamar data communications - http://www.nacamar.de - http://www.nacamar.net
KR433/KR11-RIPE - PGP-KFP = F9 A0 DF 91 74 07 6A 1C  5F 0B E0 6B 4D CD 8C 44

Mime
View raw message