www-mirrors mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason van Zyl <ja...@zenplex.com>
Subject Re: Mirror reorganization
Date Tue, 13 Aug 2002 18:39:02 GMT
On Tue, 2002-08-13 at 13:53, Aaron Bannert wrote:
> On Mon, Aug 12, 2002 at 11:07:44AM -0700, Aaron Bannert wrote:
> > If having to download older distributions presents a bandwidth problem
> > to our mirrors, then maybe we should think about a naming scheme that
> > doesn't cause our mirros to re-download the same files once they move
> > from the main distribution directory to the old/ directories, since that
> > effectively doubles the amount of bytes downloaded to the mirrors. Maybe
> > a "current" symlink to a directory of the correct version.
> Just to reiterate how much of a problem I think this is, last night
> in just the httpd-site directory, my mirror downloaded around 250MB
> of data, merely because the older binary distributions were moved to
> a new directory. This negates much of the efficiency of rsync.
> Therefore I propose that we restructure the directory layout to avoid the
> need to move files once they are on the site, use web pages to provide
> the proper amount of browsability and indirection, and at the same time
> come up with a structure that allows for efficient load-balancing to
> the mirrors:

How encapsulating all the artifacts for a project under it's project id?

This is what I've done with the Maven repository that's being mirrored @


I think this makes it easier for people to find project artifacts and
then within that directory sort the artifacts into directories by type.
So far in the Maven we have jars and distributions. I think the
everything grouped by project id is easier to navigate and manage. This
has been my experience so far trying to deal with the Maven repository.

> 1) each project places source and binary distributions under
>    "dist/projectname/"
> 2) The "projectname" directory contains an index.html listing
>    the current source and binary distributions, links for
>    any subproject, and links to the main project pages. Release
>    notes are also appropriate for this page. The main current
>    distribution is also prominently linked from this page.
> 3) Source files are placed in a src/ directory, with the following
>    naming convention [only a proposal at this point], and are not
>    moved once in place:
>      src/projectname-version-src.tar.gz
>   or src/projectname-version-src.zip
> 4) Binary distributions are placed in a binaries/ directory, with
>    each distribution having its own subdirectory. Once files are
>    in place, they are not moved. For example:
>      binaries/linux/httpd-2.0.40-i686-pc-linux-gnu.tar.gz
>   or binaries/win32/httpd-2.0.40-win32-x86.msi
> 5) All source and binary distributions are accompanied by a signature,
>    preferably made by someone on that project's PMC. The file shall
>    exist in the same location as the signed file, and have the exact
>    same name, with ".asc" appended:
>      src/httpd-2.0.40-src.tar.gz.asc
> [optional]
> 6) All source and binary distributions are accompanied by an MD5 checksum.
>    The file shall exist in the same location as the checksummed file,
>    and have the same name with ".md5" appended:
>      src/httpd-2.0.40-src.tar.gz.md5
> Key points to this proposal:
> - Files are not moved once they are in place. This saves unnecessary bandwidth
>   consumption by our mirrors when they are syncing to the main site.
> - Web pages are constructed to identify the most current distribution,
>   as well as other information pertinent to the project.
> - The structure allows us to construct intelligent load-balancing scripts
>   to distribute the load away from our central resources (daedalus).
> - The index.html files can be easily generated, so they would require
>   little or no intervention to maintain.
> -aaron

Jason van Zyl

In short, man creates for himself a new religion of a rational
and technical order to justify his work and to be justified in it.
  -- Jacques Ellul, The Technological Society

View raw message