On Mon, 18 Sep 2006, Brian Akins wrote:
> Niklas Edmundsson wrote:
>
>>
>> Extra tracking sounds unnecessary if you can do it in a way that
>> doesn't need it.
>
> It's not "extra" it just adding some tracking. When an objects gets cached
> log (sql, db, whatever) that /blah/foo/bar.html is cached as
> /cache/x/y/something.meta. Then it's very easy to ask the "store" what is
> /blah/foo/bar.html cached as? There may be multiples because of vary.
"Extra" because you already have the needed info to puzzle the things
together...
>> * Clients read from the cache as files are being cached.
>
> That's the hard one, IMO
But the implementation was rather easy once the "cache to separate
file and mv to correct location"-stuff was ripped out. Or, as easy as
building your own bucket-type is.
>> * Only one session caches the same file.
>
> Easy to do if we use deterministic tmp files and not the way we currently do
> it. Then all you have to do is when creating temp files use O_EXCL.
Or, if we skip the tmp files altogether.
>> * Header/Body updates.
>
> Eaiser with seperate files like mod_disk_cache does now.
True.
>> * No index/files out-of-sync issues. Ever.
>
> Hard to guarantee, but not impossible. Always to index when storing file and
> remove when deleting. This should use something like providers so it's not
> in core cache code and can be easily modified.
>
>> With locks, yes it's possible but also a hassle to get right with
>> performance intact.
>
> Not really that hard. Trust me it has been done...
I'll take your word for that.
>> We, as a ftp mirror operated by a non-profit computer club, have a
>> slightly different usecase with single files larger than machine RAM
>> and a working set of approx 40 times larger than RAM. Some bad design
>> decisions in mod_disk_cache becomes really visible in this
>> environment.
>
> Seems to me you should approach problem differently, like rsyncing the
> mirrored content. I don't know your environment, but was just what I cam up
> with off the top of my head.
Try rsyncing a few TB of content onto a few hundred GB of cache disk
and see how that works out for you :)
Our setup is briefly described here by the way:
http://ftp.acc.umu.se/mirror/ftp-about.html
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke@acc.umu.se
---------------------------------------------------------------------------
A closed mouth gathers no feet.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
|