httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From (David Robinson)
Subject Re: proxying
Date Mon, 22 Jan 1996 17:19:00 GMT
(I'm glad my message generated so much traffic)

Alexei wrote:
>Besides, it [a lack of inodes] may not be *that* much of a problem. Certainly
>not as much as, say, news is. An informal study (punching up about:cache in
>my copy of Netscape 2.0b5) shows the following data, which I'll take to be
>somewhat applicable, since I don't do anything extradoinary with Netscape but
>look at web pages (with images):
>Max size: 2 megs (this is what I have it set to in the Preferences)
>Current size: 1.6 megs
>Number of files: 379
>Avg size of files: 4.3k
>Now, obviously... 379 files isn't going to kill anyone's inode limits. 
>Even a 100 meg cache (more than that, and I have to wonder what's the
>point, especially given the dymnaic nature of web content) will only have
>about 25 thousand files. Our Usenet spool directory here uses 182 thousand
>inodes. (and 1.9 gigs, but that's a different story). At any rate, I vote
>inode use isn't a problem unless you have a *really* large cache, or an 
>old OS... but then, if you have an old OS... what are you doing running a 
>new web server like Apache, eh? :)

Here is a real-world implementation that we should be aiming at supporting.
(Ok, not a typical one...)

>The [UK] National Web Cache now runs on a collection of machines. None of
>these is responsible for the whole load, instead the load is evenly
>distributed between them. The machines are a single SGI Challenge DM server
>with 256 MB of RAM and 14 GB of disk and five SGI Challenge S servers each
>with 128 MB of RAM and 16 GB of disk. 

I make that a 94 Gb cache area.

Does DBM work well on 94Gb (or even 16Gb) databases?

The biggest problem with DBM (as described by sameer) is one of locking.
How does one lock only part of the database? flock(), as described by
Paul Richards, locks the entire database. This simply does not scale with
the size of the database; it is essential to be able to lock individual
database entries (aka files).

It is also true that the UNIX filesystem is not ideal for this application.
The drawbacks are the large overhead (an inode) per file, and the slow
directory search (is it linear?). However, I think that using the UNIX
fs is ultimately more portable.


View raw message