httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niklas Edmundsson <>
Subject Re: Possible new cache architecture
Date Tue, 02 May 2006 13:21:57 GMT
On Tue, 2 May 2006, Plüm, Rüdiger, VF EITO wrote:

> Another thing: I guess on systems with no mmap support the current implementation
> of mod_disk_cache will eat up a lot of memory if you cache a large local file,
> because it transforms the file bucket(s) into heap buckets in this case.
> Even if mmap is present I think that mod_disk_cache causes the file buckets
> to be transformed into many mmap buckets if the file is large. Thus we do not
> use sendfile in the case we cache the file.

Correct. When caching a 4.3GB file on a 32bit arch it gets so bad that 
mmap eats all your address space and the thing segfaults. I initally 
thought it was eating memory, but that's only if you have mmap 

> I the case that a brigade only contains file_buckets it might be possible to
> "copy" this brigade, sent it up the chain and process the copy of the brigade
> for disk storage afterwards. Of course this opens a race if the file gets
> changed in between these operations.
> This approach does not work with socket or pipe buckets for obvious reasons.
> Even heap buckets seem to be a somewhat critical idea because of the 
> added memory usage.

I did the somewhat naive approach of only doing background caching 
when the buckets refer to a single sequential file. It's not perfect, 
but it solves the main case where you get a huge amount of data to 
store ...

/Nikke - stumbled upon more than one bug when digging into
  Niklas Edmundsson, Admin @ {acc,hpc2n}      |
  Anything is edible if it's chopped finely enough

View raw message