httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niklas Edmundsson <>
Subject [PATCHES] mod_disk_cache read-while-caching
Date Thu, 05 Oct 2006 21:01:57 GMT

OK, here comes the latest two patches in the mod_disk_cache 
improvement parody. I'll attach these patches to bug #39380, but with 
less comments.

PATCH 1 - mod_disk_cache-loadstore.patch

The goal of the first patch (loadstore) is to do away with the 
write-to-file-then-move-in-place mentality.

This is done by rearranging the load (open_entity) and store 
(store_headers) into multiple functions to get better overview, and 
thus enable the possibillity to detect partial reads and other issues 
that can happen.

This also introduces a disk format change, since we need the size of 
the body in the header to be able to know what's happening.

And yes, more fixes of error handling in the helper functions.

Since diff doesn't grok what's going on the patch is rather unreadable 
by itself, I recommend applying it and looking at the result instead.

This code is based on our production code, but since we have evolved 
it into having one file for both header and body I can only say that 
it has had mild testing.

The short list of things to watch out for:
* The introduction of a new config variable, to be able to set the
   timeout for when to assume that a file update is to be seen as
   failed/old. Needs docco somewhere I suppose.
  * I'm fairly sure that I fixed a bug in the handling of
   Vary-header-rewrites. The old code rewrote the vary headers into the
   destination header file on rewrites, which would be overwritten by
   new headers in the same rewrite operation. I have no clue on how
   Vary works though, and I surely don't think our FTP server can produce
   such things.
* I'm not sure that I have gotten the logic foolprof for this
   "header/body in separate files" case. I took the route of applying
   the same logic to both header and body files. It might make more sense
   to keep the rename-in-place for headers.
* Since I'm not 100% certain on all the variations on how an object in
   the cache might be replaced I have probably missed a few testcases.
   For example, I think that rewrites occurring more often than the
   update timeout is going to be gleefully ignored. But then, why is
   the object cached in the first place?

In any case, this is the best I can do without rewriting this part 
from scratch (which I won't do, I've wasted way too much time on this 
already). It's to be seen as a necessary evil to get to the more 
interesting parts, and I can honestly say that I don't like this code 
even though it's a tad better than the original.

PATCH 2 - mod_disk_cache-rwc.patch

This is where there FINALLY is a result that is noticeable by the end 
user. Read-while-caching. Trigger a download of a huge file, that 
access will stall until the file is cached as usual. BUT, all other 
accesses will deliver what's cached so far and then "follow the 
leader" ... This does wonders (ie makes it usable) for things like a 
new linux-distro-release when you suddenly have 500 concurrent 
downloads of the same uncached DVD image.

A known problem with this patch is that the setaside-function most 
probably is incomplete. I didn't have the energy to research what it's 
really supposed to do, and since the "unfixed" cases never happens in 
our usage I didn't bother. It's probably a few minutes of coding if 
you know what it's supposed to do.

And yes, this is where I banged my head into the wall for a while 
until I figured out that someone had broken core_filters.c to do 
ReallyWrong<tm> when trying to send brigades with morphing buckets (or 
any brigade containing more than one bucket, I think).

Anyhow, you'll need the fix for that problem or you'll have the 
pleasure of a nondescriptive error message in the log and a truncated 
file delivered to the client while a file is being cached.

  Niklas Edmundsson, Admin @ {acc,hpc2n}      |
  If nobody measures up, check your yardstick.
View raw message