www-mirrors mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Bannert <aa...@clove.org>
Subject Mirror reorganization
Date Tue, 13 Aug 2002 17:53:33 GMT
On Mon, Aug 12, 2002 at 11:07:44AM -0700, Aaron Bannert wrote:
> If having to download older distributions presents a bandwidth problem
> to our mirrors, then maybe we should think about a naming scheme that
> doesn't cause our mirros to re-download the same files once they move
> from the main distribution directory to the old/ directories, since that
> effectively doubles the amount of bytes downloaded to the mirrors. Maybe
> a "current" symlink to a directory of the correct version.

Just to reiterate how much of a problem I think this is, last night
in just the httpd-site directory, my mirror downloaded around 250MB
of data, merely because the older binary distributions were moved to
a new directory. This negates much of the efficiency of rsync.

Therefore I propose that we restructure the directory layout to avoid the
need to move files once they are on the site, use web pages to provide
the proper amount of browsability and indirection, and at the same time
come up with a structure that allows for efficient load-balancing to
the mirrors:

1) each project places source and binary distributions under
   "dist/projectname/"

2) The "projectname" directory contains an index.html listing
   the current source and binary distributions, links for
   any subproject, and links to the main project pages. Release
   notes are also appropriate for this page. The main current
   distribution is also prominently linked from this page.

3) Source files are placed in a src/ directory, with the following
   naming convention [only a proposal at this point], and are not
   moved once in place:
     src/projectname-version-src.tar.gz
  or src/projectname-version-src.zip

4) Binary distributions are placed in a binaries/ directory, with
   each distribution having its own subdirectory. Once files are
   in place, they are not moved. For example:
     binaries/linux/httpd-2.0.40-i686-pc-linux-gnu.tar.gz
  or binaries/win32/httpd-2.0.40-win32-x86.msi

5) All source and binary distributions are accompanied by a signature,
   preferably made by someone on that project's PMC. The file shall
   exist in the same location as the signed file, and have the exact
   same name, with ".asc" appended:
     src/httpd-2.0.40-src.tar.gz.asc

[optional]
6) All source and binary distributions are accompanied by an MD5 checksum.
   The file shall exist in the same location as the checksummed file,
   and have the same name with ".md5" appended:
     src/httpd-2.0.40-src.tar.gz.md5


Key points to this proposal:

- Files are not moved once they are in place. This saves unnecessary bandwidth
  consumption by our mirrors when they are syncing to the main site.
- Web pages are constructed to identify the most current distribution,
  as well as other information pertinent to the project.
- The structure allows us to construct intelligent load-balancing scripts
  to distribute the load away from our central resources (daedalus).
- The index.html files can be easily generated, so they would require
  little or no intervention to maintain.

-aaron

Mime
View raw message