Date: Wed, 28 Apr 1999 11:03:47 +0200
From: "Ralf S. Engelschall" <rse@engelschall.com>
To: new-httpd@apache.org
Subject: Re: [PATCH] Extended API (EAPI)
Message-ID: <19990428110346.A38643@engelschall.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Organization: Engelschall, Germany.
Sender: new-httpd-owner@apache.org
Precedence: bulk
Reply-To: new-httpd@apache.org


IMPORTANT WARNING IN ADVANCE: 

This starts again as a technical discussion and I'm more than happy to speak
about the EAPI topic as long as you don't again try to blame me in advance for
my approaches just because it's fun and you have some nice rhetoric questions
at hand you want to push into the round. This time when I find out that I'm
again pressed into fighting for my decisions (what I don't want), I'll
immediately stop to reply. So, please be constructive and first look at the
code before you ask general things like "Why do we need an API extension at
all", "anything which isn't type-safe is useless", "strings are always bad for
keys, so EAPI is still useless", "what? a fixed set of signatures. That's
silly", etc. I'm currently a bag of nerves, so I've to force me not to discuss
too much.

In article <19990427150635.79861.qmail@zap.ne.mediaone.net> you wrote:

> I too wish it was somewhat broken into more bite sized peices which we
> could chew up before swallowing - if only to torture Ralf.

Hmmm.... where do you suggest that we break up the stuff, Ben?  Currently I
think we can break up the eapi.patch itself into only three pieces:

  1. ap/ap_{hook,ctx,mm} related configuration support patches
  2. API extensions like additional hooks, etc.
  3. memory pool related patches for shared memory pools

I think that's very reasonable. At least I suggest the following: We can
commit 1) and 2) without #ifdef EAPI's and 3) completely wrapped with
#ifdef's. Then when the shared memory pools were reviewed more carefully we
can decide on removing #ifdef's here, too. At least points 1) and 2) should be
no problem when comitted even without #ifdef's because it already survived
practice for for more than six months now inside a lot of webservers around
the world. But for point 3) I really want more reviews before I would remove
the #ifdef's.

>[...]
> Let me chew on the hook scheme for a few paragraphs.
>[...]
> My problem with promoting it to being a first class part of the the
> API is that I can't quite stomach the design as it stands.
> 
> We need to make a general ap_hook_* facility with the intent of making
> the module API more plyable.
> 
> I'd very much like to move toward a hook registry model that can
> replace the brittle set of method slots found in the module structure.
> 
> Then when new hooks are added in the core it wouldn't be such
> a big rock tossed into a small pond.
> 
> Instead modules could hook up in their init function.  In time the
> module structure might even wither into a vistigal organ.
> 
> Note that we already "compile" the methods out of the module structure
> and into something more efficent (e.g.  build_method_shortcuts in
> http_config.c).  I have trouble distinguishing the problem all that is
> trying to solve from the one that Ralf needed to solve.
> 
> The run_all flag to run_method could easily be folded into Ralf's
> ap_hook_sig.  We might even - heaven forbid - provide variations of
> ap_hook_sig that support an error protocol.
> 
> I'm sad that the method keys are strings, rather than symbols or
> atoms.  We really do need atoms, back filling their use into the
> current API is an amusing mess.

Yes, sounds very reasonable for a long-term API solution. Just keep in mind
that my current EAPI was designed to be a real _optional addon_, so I've still
not changed the old static module hooks, of course. But when we approach a
long-term solution for Apache 2.0 it should go the way you suggest.

Perhaps some background on EAPI (either try to understand it and give
constructive critics on how we can make it better or shut up, please):

1. Strings instead of numerical symbols as the keys:

The point of using strings instead of symbols for the keys is correct: it
could be technically optimized, as Dean also already pointed out. The reason
why I've used strings were just that for strings I need no explicit temporary
storage or initialization/declaration/mapping. This was an important design
goal for EAPI because adding hooks to the existing modules had to be reduced
really to a minimum.  So strings were the most convinient keys here, because
they are handled by the compiler automatically at both producer _AND_ consumer
side of the EAPI hook mechanism and still provide an intuitive name (numbers
like 1, 2, 3 are not of this type and for an HOOK_FOOBAR is need a header file
shared by consumer and producer). Whenever you use some other type of symbols
you either loose the intuitive names or have to declare name mappings at
two(!) sides. 

2. The restrictions:

The EAPI hook mechanism is damn cool when it comes to programming and patching
existing things, because your patching is reduced to a minimum and the
mechanism is _very_ flexible and this way one can use it to program arbitrary
extensions as experience showed (for instance Paul was able to program large
extensions for mod_proxy totally _outside_ of mod_proxy without much hacking
_inside_ mod_proxy. OTOH the current ugly hack for #perl in mod_include which
breaks DSO could be replaced by a clean EAPI solution which would work even
with DSO).  Nevertheless the EAPI hook mechanims has still two faults: it
doesn't (and cannot) provide type-safeness and useable function signatures are
restricted to a pre-implemented set and isn't (and cannot) totally arbitrary. 

2.a) The hook mechanism isn't type-safe:

Let me explain the reasons: The type-safeness again has to do with the
technical restrictions related to the producer/consumer side.  Whenever you
really want a totally loosly coupled hook mechanims (where there isn't _ANY_
API information passing between producer and consumer, i.e. not even a common
header file or redundant specification of the details!) you cannot provide
type-safeness IMHO in ANSI C. I've discussed this very long with Martin K. and
other people who know ANSI C very well, but we always came to the conclusion
that with plain ANSI C one _cannot_ implement both the loosly coupled facility
_and_ type-safeness at the same type.  The only chance would be some sort of
special pre-processor run before the compiler run, but that idea we abbandoned
because it's overkill and ugly. So I decided to accept that type-safeness in
the hook calls isn't available for the advantage that the hook mechanism is
really totally loosly coupled and this way works _smoothly_ with technical
restrictions like DSO and even politicial restrictions (like patching
reduction).

BTW, I've some EAPI-variant on my disk from a contributor who tried to make it
type-safe. And as you can guess it both violates the idea of the 100%
loosly-coupled mechanism and is really a _HORRIBLE_ mess of pre-processor
hacks. I've thought long about this variant and whether I should overtake it
(after cleaning up, of course), but I've at the end found it not maintainable.

2.b) The arbitrary but fixed set of implemented signatures:

The fixed set of implemented function signatures again have to do with this:
When you want it loosly coupled the facility itself without any redundancy in
specifying details, only at one location the signature is specified and it
then has to be store in the hook mechanism itself.  But unlike scripting
languages there isn't an "eval" in ANSI C, so you cannot "create" a new
function call very easily _and_ portable out of the arguments on the stack. We
again here discussed a lot (especially with Martin) and the only chance in
ANSI C we found is to use a combination of varargs for passing the signature
from the caller to the hook mechanism and hard-coded/dispatched calls based on
va_arg from the hook mechanism to the target function. 

The problem is that in ANSI C you don't have a va_arg() without specifying the
type of the object (because ANSI C doesn't know the size of the objects on the
stack). That's why ap_hook.c implements a fixed set of signature-related
calls. In practice that's usually no problem, because one needs only a few
combinations and the Perl script appended to ap_hook.c does the job for
creating the dispatch list. The great advantage is that one still uses only
plain ANSI C and no compiler-dependent tricks or even assembler stuff.  This
was more important for me than allowing totally arbitrary function signatures,
so I've decided to go this way.

Greetings,
                                       Ralf S. Engelschall
                                       rse@engelschall.com
                                       www.engelschall.com