OK, here comes the latest two patches in the mod_disk_cache
improvement parody. I'll attach these patches to bug #39380, but with
less comments.
---------------------------------------------
PATCH 1 - mod_disk_cache-loadstore.patch
The goal of the first patch (loadstore) is to do away with the
write-to-file-then-move-in-place mentality.
This is done by rearranging the load (open_entity) and store
(store_headers) into multiple functions to get better overview, and
thus enable the possibillity to detect partial reads and other issues
that can happen.
This also introduces a disk format change, since we need the size of
the body in the header to be able to know what's happening.
And yes, more fixes of error handling in the helper functions.
Since diff doesn't grok what's going on the patch is rather unreadable
by itself, I recommend applying it and looking at the result instead.
This code is based on our production code, but since we have evolved
it into having one file for both header and body I can only say that
it has had mild testing.
The short list of things to watch out for:
* The introduction of a new config variable, to be able to set the
timeout for when to assume that a file update is to be seen as
failed/old. Needs docco somewhere I suppose.
* I'm fairly sure that I fixed a bug in the handling of
Vary-header-rewrites. The old code rewrote the vary headers into the
destination header file on rewrites, which would be overwritten by
new headers in the same rewrite operation. I have no clue on how
Vary works though, and I surely don't think our FTP server can produce
such things.
* I'm not sure that I have gotten the logic foolprof for this
"header/body in separate files" case. I took the route of applying
the same logic to both header and body files. It might make more sense
to keep the rename-in-place for headers.
* Since I'm not 100% certain on all the variations on how an object in
the cache might be replaced I have probably missed a few testcases.
For example, I think that rewrites occurring more often than the
update timeout is going to be gleefully ignored. But then, why is
the object cached in the first place?
In any case, this is the best I can do without rewriting this part
from scratch (which I won't do, I've wasted way too much time on this
already). It's to be seen as a necessary evil to get to the more
interesting parts, and I can honestly say that I don't like this code
even though it's a tad better than the original.
---------------------------------------------
---------------------------------------------
PATCH 2 - mod_disk_cache-rwc.patch
This is where there FINALLY is a result that is noticeable by the end
user. Read-while-caching. Trigger a download of a huge file, that
access will stall until the file is cached as usual. BUT, all other
accesses will deliver what's cached so far and then "follow the
leader" ... This does wonders (ie makes it usable) for things like a
new linux-distro-release when you suddenly have 500 concurrent
downloads of the same uncached DVD image.
A known problem with this patch is that the setaside-function most
probably is incomplete. I didn't have the energy to research what it's
really supposed to do, and since the "unfixed" cases never happens in
our usage I didn't bother. It's probably a few minutes of coding if
you know what it's supposed to do.
And yes, this is where I banged my head into the wall for a while
until I figured out that someone had broken core_filters.c to do
ReallyWrong<tm> when trying to send brigades with morphing buckets (or
any brigade containing more than one bucket, I think).
Anyhow, you'll need the fix for that problem or you'll have the
pleasure of a nondescriptive error message in the log and a truncated
file delivered to the client while a file is being cached.
---------------------------------------------
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke@acc.umu.se
---------------------------------------------------------------------------
If nobody measures up, check your yardstick.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
|