perl-docs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stas Bekman <s...@stason.org>
Subject Re: Mirrors
Date Tue, 02 Jul 2002 09:34:40 GMT
Thomas Klausner wrote:
> Hi!
> 
> On Tue, Jul 02, 2002 at 10:17:37AM +0200, Per Einar Ellefsen wrote:
> 
>>Thomas has said he wanted to do one in Austria (that's the country, 
>>right?).
> 
> Yes, that's what's behind .at
> 
> Rather small (~8 million people), between German and Italy,
> Switzerland and Hungaria, Czech & Slovak Republic...
> 
> 
>>That makes it one for Europe. I'm sure we can get more, for 
>>example I'm currently looking at sunsite.dk which seem to have hosted a 
>>mirror for the XEmacs site and others (they have the apache.org website), 
>>so that'd make for another one in Europe. Other continents might appear to 
>>us in the future.
>>
>>If we have *total control*, is everyone ok with proceeding with this?
> 
> 
> I'm planning to write two scripts
> 
> 1) build-mirror:
> 
> This script will be run on each mirror via a cronjob. Frequency should
> be definitly not more than weekly, better dayly.
> 
> *) fetches the docs from cvs
> *) builds them with DocSet
> *) runs the search indexer (using SWISH)
> 
> Sites running this script will need DocSet, cvs, SWISH-E
> installed. This should be a requirement for all "official" mirrors

i actually have these crontabs already:

this one forces a complete rebuild, which is quite resource intensive 
because of the pdfs (needed when templates change)

/home/perlwww/bin/site_build_force:
-----------------------------------
#!/bin/sh

umask 002

PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/bin:/usr/X11R6/bin:$HOME/bin; 
export PATH
PERL5LIB=$HOME/lib/perl5/5.00503:$HOME/lib/perl5/site_perl/5.005:$HOME/lib/perl5/site_perl:$HOME/lib/perl5;

export PERL5LIB

cd /home/perlwww/apache.org/modperl-docs
cvs up -dP >/dev/null 2>&1
bin/build -df
bin/makeindex

this one is similar but does only updates (suitable on doc updates)

/home/perlwww/bin/site_build_update:
-------------------------------------
#!/bin/sh

umask 002

PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/bin:/usr/X11R6/bin:$HOME/bin; 
export PATH
PERL5LIB=$HOME/lib/perl5/5.00503:$HOME/lib/perl5/site_perl/5.005:$HOME/lib/perl5/site_perl:$HOME/lib/perl5;

export PERL5LIB

cd /home/perlwww/apache.org/modperl-docs
cvs up -dP >/dev/null 2>&1
bin/build -d
bin/makeindex


crontab -l
# every monday rebuild all, including pdf
30 05  * * 1 /home/perlwww/bin/site_build_force
# update all (only changes) every 6 hours
15 */6 * * * /home/perlwww/bin/site_build_update


> 2) test-mirrors
> 
> This script will be run on perl.apache.org via a cronjob. Should be in
> sync with the building there, maybe even in the same script
> 
> *) request /index.html from each mirror (LWP)
> *) check the timestamp
> *) if last-modified is recent enough (say, 1 week) add to mirrors

you mean the opposite, no? when not recent enough, remove from the 
mirrors list.

> *) add each mirror to a (new) Mirrors-DocSet
> 
> What do you think?

there is a problem with any automatic modification, since they require 
auth to commit changes. currently the online site is checked out as an 
anonymous user: pserver:anoncvs@cvs.apache.org:/home/cvspublic

another reason for not doing *any* changes on the apache.org site is to 
avoid colissions with automatic updates. if nobody changes things there, 
we can be sure that 'cvs up' will never render the documents broken.

I'd rather have this script maintain a black-list which we will manually 
apply to our list every so often.

what if the mirror updates the first page but fails to do the same for 
other pages?

do we require that the search works locally? and if so should we verify 
that the indices are up to date?


__________________________________________________________________
Stas Bekman            JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/     mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org   http://ticketmaster.com



---------------------------------------------------------------------
To unsubscribe, e-mail: docs-dev-unsubscribe@perl.apache.org
For additional commands, e-mail: docs-dev-help@perl.apache.org


Mime
View raw message