httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colm MacCarthaigh <>
Subject Re: [PATCH] fix incorrect 304's responses when cache is unwritable
Date Sun, 07 Aug 2005 23:04:44 GMT
On Mon, Aug 08, 2005 at 12:07:09AM +0200, Andreas Steinmetz wrote:
> Colm MacCarthaigh wrote:
> > For what it's worth though, htcacheclean itself has this massive bug,
> > and does not do any directory cleanup, so your patch isn't alone in
> > doing this.
> The problem is that you can't remove directories with htcacheclean
> without generating race conditions wrt. httpd.
> Assume that htcacheclean removes the last entries from a directory and
> then removes the directory. At the same time httpd wants to use the
> directory as it was already there...

Well that's a pretty each race to solve within httpd. It won't be able
to create the headers, or the body. The patch I've submitted cleans up
that slight race. The file won't be cached on that serve, but I don't
think that's a big deal :-)

> You'll be better off setting CacheDirLength and CacheDirLevels to
> sensible values. Try:
> CacheDirLength 1
> CacheDirLevels 2
> You can get at most 64^2 directories which is 4096 directories.

But when the link count for the directories themselves will get very
large indeed, and common filesystems like ext3 or XFS will get pretty
slow indeed. The whole point of allowing them to be split up is to avoid
this :)

>  Any
> reasonable filesystem can stand that. Now assume an average of 500
> cached objects per directory which the filesystem should easily manage.
> I tend to believe that 2048000 cached objects is quite a lot. If the
> size of these objects has an average of only 1KB you already have a
> total cache size of 2GB.

Well I run a cache which is 135Gb, which regularly has nearly 50 million
jpegs in it (revisions of satelite imagery of the entire planet, not
porn), but my numbers are always insane. For another example;

I turned on the cache only about 2 hours ago (after applying the patch
I've sent) and it's already at 537,029 cached files. (though it fills in
spurts, as other projects pull from us).

> Even if you tried 3 for CacheDirLevels it would be 262144 directories
> maximum which should be still fine for any reasonable filesystem.

You're right, and I'm being a bit over the top with my numbers and
examples, but I still think there is danger of inode exhaustion in
real-world configurations. 

I've run proxy clusters in the past which handled tens of millions of
requests per day. I don't see why a cluster of Apache 2 proxies
shouldn't be able to share a network accessible mod_disk_cache cache
area, for example.

Colm MacCárthaigh                        Public Key:

View raw message