Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@httpd.apache.org
Received-SPF: neutral (herse.apache.org: local policy)
Date: Wed, 14 Feb 2007 15:37:01 -0800
From: Chris Darroch <chrisd@pearsoncmg.com>
Subject: Re: 3.0 - Proposed Goals
In-reply-to: <45D2BB47.10507@force-elite.com>
To: dev@httpd.apache.org
Message-id: <45D39D1D.2010006@pearsoncmg.com>
Organization: Pearson CTG/CMG
MIME-version: 1.0
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7bit
References: <45D2BB47.10507@force-elite.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.13) Gecko/20060423

Hi --

Paul Querna wrote:

> - Rewrite the Core to be an Async Event state machine and data router.
> - Break the 1:1 mapping of a worker to a single request.
> - Change the meaning of MPMs. The problem with MPMs today is they are
> really mostly platform abstractions -- not just abstractions of the
> process model itself.
> - Build a cleaner configuration system, enabling runtime
> reconfiguration.
> - Experiment with the right way to abstract state machines,
> multi-threading, and async IO from module developers who want a 'simple
> world view'.

   At a high level, I like these goals!  I'd agree with others that
it's probably most important to set specific goals which are achievable
in a mid-term timeframe (3-6 months?) and focus on those.  The process
of actually implementing them is likely to spur new ideas, as well.

   I'm not keen on requiring an XML configuration file format; so long
as it's optional, that's OK with me.

   Here are a few itches that I'd personally like to see scratched as
well; they may or may not align with your proposals:

- Stop passing sub-requests through the potentially expensive authn/z
  steps if they share the same authn/z configuration as the main
  request.  This would have security implications for certain external
  modules like mod_authz_svn, which currently expect to be called to do
  private authorization of each sub-request.  However, there are otherwise
  some serious performance implications when using, say, mod_dav
  with mod_authn_dbd.[1]

- Provide a generic inter-process data-sharing framework.  Currently
  mod_ssl, mod_auth_digest, mod_ldap, and the scoreboard all use
  more-or-less independent implementations of shared memory data stores.
  As someone who maintains a module with yet another such data store,
  I think a standard interface for such things (beyond apr_rmm) might be
  useful.  Perhaps something key/value based; maybe aligned with memcached
  somehow?  See my final musings below.

- Provide a generic scoreboard interface for use by modules.  The
  current scoreboard is effectively sized at initial startup to
  max MPM processes * max MPM threads.  That wastes space, but also
  provides no way for modules to register their own private threads.
  As someone who maintains a module with such threads, I'd love to
  see them in mod_status.  I'd also like to see the non-worker threads
  from an MPM like worker in there too (i.e., listener, start, main);
  I have a collection of incomplete patches to do this hanging around.

   Admittedly, this may be a hard problem: how do you size the scoreboard's
block of shared memory if modules can be added at restarts, and might
suddenly require extra scoreboard space for their threads?  I have no good
solution.  (Should the scoreboard use the above-mentioned generic
data-sharing framework, or not?  Perhaps shared memory isn't even the
right tool?)

   If we're generally moving to an increasingly asynchronous, threaded
design then I think such a scoreboard might also serve as a valuable
sanity check during development ("What the heck is that thread doing?")

   An API which allowed threads to register their possible states might
be valuable; this would allow modules/providers/MPMs to define what
states were meaningful to them, rather than trying to define them
all in scoreboard.h.

   I confess I haven't followed the progress of the httpd-proxy-scoreboard
branch; maybe there's some work in there that would apply to these issues.

   As a long-term goal I think it would be interesting to try to design
these interfaces in such a way as to allow them to work between multiple
instances of httpd.  This obviously heads into the tricky territory of
distributed computing, clustering, etc.  If one can't permit stale or
cached data you may need write replication, a distributed lock manager,
leader election schemes, and so forth.

   This is obviously complex stuff and maybe out of scope for httpd.
Still, I can't help but feel like there's a logical continuum here
from the existing httpd 2.x shared memory data stores, to memcached,
to a distributed locking and data storage system like Google's Chubby
lock service.[2]

   Moving httpd's existing uses of inter-process data stores to a
generic key/value interface might allow us to start with just a
default provider that had a shared memory implementation no different
than today's.  Other providers could then be developed later to
replicate the data across a cluster, if so desired.

Chris.

[1] http://marc.theaimsgroup.com/?l=apache-httpd-dev&m=116556310814307&w=2
[2] http://labs.google.com/papers/chubby.html

-- 
GPG Key ID: 366A375B
GPG Key Fingerprint: 485E 5041 17E1 E2BB C263  E4DE C8E3 FA36 366A 375B