httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niklas Edmundsson <>
Subject Re: mod_cache responsibilities vs mod_xxx_cache provider responsibilities
Date Wed, 20 Sep 2006 08:06:20 GMT
On Mon, 18 Sep 2006, Brian Akins wrote:

> Niklas Edmundsson wrote:
>> Extra tracking sounds unnecessary if you can do it in a way that
>> doesn't need it.
> It's not "extra" it just adding some tracking.  When an objects gets cached 
> log (sql, db, whatever) that /blah/foo/bar.html is cached as 
> /cache/x/y/something.meta.  Then it's very easy to ask the "store" what is 
> /blah/foo/bar.html cached as?  There may be multiples because of vary.

"Extra" because you already have the needed info to puzzle the things 

>> * Clients read from the cache as files are being cached.
> That's the hard one, IMO

But the implementation was rather easy once the "cache to separate 
file and mv to correct location"-stuff was ripped out. Or, as easy as 
building your own bucket-type is.

>> * Only one session caches the same file.
> Easy to do if we use deterministic tmp files and not the way we currently do 
> it.  Then all you have to do is when creating temp files use O_EXCL.

Or, if we skip the tmp files altogether.

>> * Header/Body updates.
> Eaiser with seperate files like mod_disk_cache does now.


>> * No index/files out-of-sync issues. Ever.
> Hard to guarantee, but not impossible.  Always to index when storing file and 
> remove when deleting.  This should use something like providers so it's not 
> in core cache code and can be easily modified.
>> With locks, yes it's possible but also a hassle to get right with
>> performance intact.
> Not really that hard.  Trust me it has been done...

I'll take your word for that.

>> We, as a ftp mirror operated by a non-profit computer club, have a
>> slightly different usecase with single files larger than machine RAM
>> and a working set of approx 40 times larger than RAM. Some bad design
>> decisions in mod_disk_cache becomes really visible in this
>> environment.
> Seems to me you should approach problem differently, like rsyncing the 
> mirrored content.  I don't know your environment, but was just what I cam up 
> with off the top of my head.

Try rsyncing a few TB of content onto a few hundred GB of cache disk 
and see how that works out for you :)

Our setup is briefly described here by the way:

  Niklas Edmundsson, Admin @ {acc,hpc2n}      |
  A closed mouth gathers no feet.

View raw message