httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niklas Edmundsson <>
Subject Re: mod_cache: store_body() bites off more than it can chew
Date Mon, 06 Sep 2010 09:52:12 GMT
On Fri, 3 Sep 2010, Graham Leggett wrote:


>> Regarding the issue of the disk cache cramming the entire file into 
>> memory/address space, an alternate solution could be that the cache returns 
>> buckets pointing to the cached file, ie that the cache consumed those pesky 
>> mmapped buckets. This way the cache could cache the file rather quickly 
>> independent of the speed of the client, so that the file caching is 
>> finished in a sane time so others can benefit from it even though the 
>> cache-initiating request is running at snail-speed...
> Been keen to do this for a while, this would definitely solve the RAM 
> problem, but wouldn't solve the time problem. Copying 4GB of data from a slow 
> disk can easily take minutes, and when Blu-ray images start becoming common, 
> the problem would get worse.

Yup. The next step to solve that would be to be able to serve requests 
from cache while they are being cached. I don't know the RFC 
implications of doing this, but in real life it's really useful.

For those who have forgotten, that's what we do in our 
large-file-caching-patchset for mod_disk_cache (hidden as an 
attachment to 
but I should really get around to upload an up2date version that 
applies cleanly to the current 2.2 release). Some of the solutions 
there aren't really applicable to httpd proper (mostly workarounds for 
missing infrastructure), but some ideas are rather sane (like writing 
the header files in a single go with an iovec with null terminated 
strings instead of crlf-stuff thad needs to be parsed). Oh, and the 
design caters for a shared data cache (ftp and rsync access uses the 
same cache), which isn't really a priority for something in httpd 

Sadly I haven't had any time to whack this into pieces and feed the 
appropriate parts into httpd trunk, but I can at least vouch for the 
fact that the functionality is really useful and essential when 
dealing with large files (ie dvd/bluray images) and that development 
in this direction is desired.

  Niklas Edmundsson, Admin @ {acc,hpc2n}      |
  Must Go - Some Jehovahs witnesses need shouting at.

View raw message