httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niklas Edmundsson <>
Subject Re: mod_cache responsibilities vs mod_xxx_cache provider responsibilities
Date Sat, 16 Sep 2006 07:23:44 GMT
On Fri, 15 Sep 2006, Brian Akins wrote:

> The separate header and body files work wonderfully for performance (filling 
> multiple gig interfaces and/or 30k requests/sec. or rather modest hardware). 
> If you have them all in one, it can make the sendfile for the body 
> cumbersome.

If you write to the file using mmap on linux, then sendfile() breaks 
yes. mmap didn't give any major performance benefit for the body copy 
though, so it doesn't matter and we don't use it. This is really a 
Linux bug, since non-overlapping write/sendfile should be OK.

> If you somehow track what "entries" or in the cache, it is very easy to purge 
> entries.

Extra tracking sounds unnecessary if you can do it in a way that 
doesn't need it.

> At Apachecon, I'll talk some about our version of mod_cache. 
> Unfortunately, I can't share code :( But I can tell you the separate 
> files way is not a performance or housekeeping issue.

If you have the index i can agree.

However, I don't see how you can do a lockless design with multiple 
files and an index that can do:

* Clients read from the cache as files are being cached.
* Only one session caches the same file.
* Header/Body updates.
* No index/files out-of-sync issues. Ever.

With locks, yes it's possible but also a hassle to get right with 
performance intact.

The current mod_disk_cache seems to be designed for small files and 
enough memory to hide the problems by the design. If you have files 
that fit into the OS cache then it doesn't matter if hundreds of 
sessions are caching the same file, it'll work out eventually without 
reduced performance. This isn't the case when each file (DVD image) is 
bigger than your memory and doesn't fit in the OS file cache. In fact 
you can tell that the author never even consider this due to the way 
the body is copied (on 32bit you loose).

We, as a ftp mirror operated by a non-profit computer club, have a 
slightly different usecase with single files larger than machine RAM 
and a working set of approx 40 times larger than RAM. Some bad design 
decisions in mod_disk_cache becomes really visible in this 

  Niklas Edmundsson, Admin @ {acc,hpc2n}      |
  I wish I had a snappy Trek Message to put here...

View raw message