httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Darroch <chr...@pearsoncmg.com>
Subject slotmen API thoughts
Date Wed, 31 Dec 2008 20:12:09 GMT
Hi --

   First of all, many thanks to Jim Jagielski, Jean-Frederic Clere,
and Brian Akins for the slotmem API!  Personally I think it would be
great to see the mod_auth_digest and/or mod_proxy module gradually migrate
toward using either the slotmem or socache APIs, as appropriate (and
perhaps even the scoreboard itself?)


   A few initial thoughts, in two categories, specific and general.

   In the specific realm, I think mod_sharedmem.c should not be
hard-coded to persist data into *.slotmem files in a "logs/" directory.
I know that we install some httpd instances with non-default layouts,
e.g., the "GNU layout" where log files live under var/apache/log and
other runtime files under var/apache/run.

   So I'd recommend that mod_sharedmem.c should support configuration
directives that allow the admininstrator to choose (a) whether to
persist data at all, (b) the filename format to use, and (c) where
to store the data files.

   Also in the specific category, I feel we should probably move
mod_slotmem.h up into include/, akin ap_socache.h and others.


   In the general category, I am somewhat unsure about the slotmem
API design, especially in relation to the socache API.  These
thoughts came up because I began thinking about how to add additional
slotmem providers and what these might look like.  Could there be
a slotmem provider that used a DB or ZooKeeper backend, for example?

   I came the conclusion that the API more or less prevented such
options, because access to slots is performed through pointers to
physical memory.  The mem method returns such a pointer given an
index.  In fact, beyond the two existing providers -- unshared and
shared memory -- I'm not at all sure what other kinds of things
one could really do.

   I suppose one could write a mod_sharedmem variant that persisted
data to a DB -- but then you're either duplicating all the code
just to adjust the persistence mechanism, or adding some kind of
sub-provider API to support different persistence choices.


   That led me to the following ideas, which I realize are somewhat
radical alterations of the initial implementation.  I'd be very
happy to contribute to such modifications, though, if others felt
they had value.  In summary:

- model API closely on socache API

- include/ap_slotcache.h defines the following:

  - does not define internals of ap_slotcache_t; leave that to providers

  - no lock/unlock functions; leave that to callers but signal
    need for them with AP_SLOTCACHE_FLAG_NOTMPSAFE (parallel to
    AP_SOCACHE_FLAG_NOTMPSAFE, note "NOTMPSAFE" == "not MP-safe")

  - slotcache provider methods:

    - create()   - takes num_items and item_size
                 - to be called during initial config pass, providers
                   should not initialize or create mutexes here, etc.
    - init()     - to be called during post-config pass, passed pconf pool,
                   performs real initialization
    - destroy()  - to be called from pconf pool cleanup
    - store()    - takes index, value of item_size length
    - retrieve() - takes index and unsigned char* buffer to write into
    - reset()    - takes index, clears value
    - maybe do(), or status(), akin to socache's status()?

  - AP_SLOTCACHE_PROVIDER_GROUP ("slotcache") and
    AP_SLOTCACHE_PROVIDER_VERSION ("0")

- drop mod_slotmem.c and mod_plainmem.c (which, from one point of
  view, is something most programmers could do themselves pretty
  simply with just a big apr_palloc())


   Finally (possibly in the bikeshed category), might I suggest
a naming scheme that follows the socache format?  Something like:

include/ap_slotcache.h
modules/cache/mod_slotcache_shm.c
modules/cache/mod_slotcache_*.c


   The default provider would likely be mod_slotcache_shm.c which
works pretty much like the existing mod_sharedmem.c, with config
directives allowing the admin to control whether to persist data
and where to put the resultant files, if desired.

   To my mind, the great advantage of having accessor methods like
store(), retrieve(), and reset() instead of a mem() method is that
a wide variety of providers can be envisioned.

   Some of these would be tuned for performance -- in particular,
mod_slotcache_shm would work basically like the scoreboard, with slab
of shared memory.  It would use the NOTMPSAFE flag to indicate that
callers need locking to ensure atomic reads/writes.

   Some users might want to add locking, but others, if they knew
they didn't care about that, could skip it.  That's how the scoreboard
works at the moment, IIRC; it skips locking in favour of fast access.
It could potentially migrate to this provider, as could the lbproxy
stuff.  (I'm not sure whether the currently-disable auth digest cache
would work better with a slot or small-object key/value cache framework.)

   Other providers could offer a wide range of backend mechanisms,
from DBM files to relational DBs to ZooKeeper and so forth.


   Thoughts?  I know this is another of my long lists of "helpful"
suggestions; my apologies for that.  I should stress again that I'm
very excited by the possibilities opened up by the slotmem API!
 
   I just feel like most of these options are currently made impossible
by the necessity of handing out memory addresses to callers -- which
effectively means allocating a big slab of memory (shared or unshared)
to cache the data.  Thus, as I noted, I'm unsure what else one could
really do beyond the two existing providers.  If we switch to
get/set/reset accessor methods, though -- like APR tables and hashes, the
socache API, etc. -- then a whole range of options seems possible!
Hence, that's my key suggestion.

   Thanks again very much for getting this into trunk!

Chris.

-- 
GPG Key ID: 366A375B
GPG Key Fingerprint: 485E 5041 17E1 E2BB C263  E4DE C8E3 FA36 366A 375B


Mime
View raw message