perl-docs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gerald Richter" <>
Subject Autogenerating XS code and Docs
Date Wed, 12 Jun 2002 07:35:21 GMT

as some of you may know, last fall I have started a project that splits out
the XS autogenerating stuff from mod_perl 2.0 and generalize it to make it
useable also outside of mod_perl. I have converted the C parser from C::Scan
to a more flexible approach with Parse::RecDescent (which needs no C
Compiler anymore), abtracted the Apache/mod_perl specific stuff into it's
own subclass and added some other features. The result runs as
ExtUtils::XSBuilder. I use this currently to build the Perl <-> C interface
of Embperl 2.0 and for generating a mod_dav 1.0 interfaces (which I hope to
release soon). Then, of course, I have tried to use it for mod_perl itself,
but due to my limited time the process of makeing it run with mod_perl has
been stalled since some time. I had a mail exchange with Stas some time ago
and we though it will be the best to make the effort public, so maybe
somebody can join in. Also I have some time now available, so I want to
continue to work on it. There are two parts:

1.) generating code
2.) generating docs

While the first requires some knowledge of XS and mod_perl / Apache
internals, the second could be done without such a need. My current version
already splits out the comments that included in the Apache header files, so
we end up with something like:

'perl_name' => 'apr_table_add',
'comment_parsed' => {
    'doxygen_param' => [
    'doxygen_param_desc' => {
                              'val' => 'The value to add.',
                              'key' => 'The key to use',
                              't' => 'The table to add to'
    'func_desc' => 'Add data to a table, regardless of whether there is
another element with the same key.',
    'doxygen_remark' => 'When adding data, this function makes a copy of
both the key and the value.'
more stuff, like mapping form C to Perl arguments etc. follows here....

Since the whole Apache API is well documented in this way, it should be
possible to autogenerate most of the mod_perl API documentation from this
informations. It would be necessary that somebody writes a tool that takes
this informations, brings it together with some manualy maintained pod and
generates the pods out of it. This shouldn't be to hard to do.

For the XS genaration stuff there some issue with pTHX_ parameters left,
which I hope to resolve soon. Also the whole stuff for generating constants
is missing yet.

I would be very happy if anybody likes to join this project, either the docs
part or help me with the XS part!

I have put the current version on my ftp server
(, ExtUtils-XSBuilder is the
general module, mpbuilder contains the mod_perl specify classes and wrapxs
is what is generated for mod_perl right now. The pdd files inside of wrapxs
contains the parsed comments. The ExtUtils-XSBuilder also contains a
XSBuilder.pod, which describes how the current stuff works, I append it for
convenience below. Also I append a list of changes that I have made, since I
have forked off from mod_perl.



- structs that are contained in another struct are now handled correctly.
  When reading the struct memeber a new object is returned, which can be
  to access the members of the contained struct. The contained struct cannot
  be set directly.

- In class name in can end with :: to use single level
  (e.g. request_rec * | Apache::  for mod_perl 1.x)

- Argspec of can specify argument as return only by
  it with < e.g.
    dav_open_lockdb | | r, ro, <lockdb
  will be called as ($error get the return value of the C function)
    ($error, $lockdb) = $r -> open_lockdb (0) ;
  The return argument (e.g. lockdb) will always be passed by address
  to the function.

- xs/xxxx_sv_convert.h now also contains macros to convert C data types to
  for non object types e.g. IV

- better handling for structs that for which a typedef doesn't exist.

- method checkmaps & writemaps creates new_*.map with all
  functions/structures/types/callbacks that are not already in a map file.

- ParseSource now scans for callbacks and generates a
  with similar format as

- WrapXS generates callbacks for struct members as method calls when the
  struct doesn't contains any field where to store some userdata. An
  version which passes the perl object in some user data field need still to
be done.

- Extra include files now given in xs/maps/foo_types instead of hardcoded

- prefixes for conversion macros and other generated functions are now
  set via an overrideable method

- support for passing structure by value as function argument / return type

- macros in xxx_sv_convert.h now correctly convert NULL to undef and

- Structure members can have a different name in Perl and C. Controlled via
  the map file.

- %convert_alias  has been moved into

- typemap ids (e.g. T_APACHEOBJ) has been moved into

- it possible to configure the way how memory for strings is allocated
  via map files, by giveing a malloctype in type map file and allocation
  code in structure map file.

- a dispatch argspec can now given in the map file, this allows to pass
  $r to the xs function and r -> pool to the C function by writing:

  ap_make_array | ap_make_array(r->pool, nelts, elt_size) | request_rec *:r,
nelts, elt_size

- handle conversion from package name to directory correctly when more then
  levels of namespaces e.g. Apache::DAV::Resource ->


=head1 Basics

ExtUtils::XSBuilder is a set modules to parse C header files and create XS
glue code and documentation out of it. Idealy this allows to "write" an
interface to a C library without coding a line. Since no C-API is ideal,
some adjuments are necessary most of the time. So to use this module you
must still be familar with C and XS programming, but it removes a lot of
stupid work and copy&paste from you. Also when the C API changes, most
of the time you only have to rerun XSBuilder to get your new Perl API.

The creation process takes place in the following steps:

=over 4

=item Derive a class from ExtUtils::XSBuilder::ParseSource

This class must override some methods to tell XSBuilder which C header files
to parse and some other necessary parameters. You need at least to override
methods C<package> to give the name of the package you want to create and
either the method C<find_includes> and return all C header files to parse or
the method C<include_dirs> to return a list of all directories which should
be scanned for C header files.

Of course there are more methods you can overide. See
L<ExtUtils::XSBuilder::ParseSource> for a list of overrideable methods.

=item Scan the source files

If your derived class is called MyClass::ParseSource you simply start the
source scan with

    perl -MMyClass::ParseSource -e 'MyClass::ParseSource->run'

You may also put this into a small script to ease usage and maybe set the
Perl libpath etc.

After you run the source scan XSBuilder has created a set of tables, which
contains the result of the parseing. If you don't have changed the defaults
in your class, the tables are created under C<xs/tables/current> and then
the name of the module you want to create as given with the method
There will be a C<> which holds all the function
a C<> which holds the structures, a C<>,
contains alls constants found in the header files and a C<>
which contains definitions for callback types.

The main reason why we not directly create XS code, but first these
tables is, that source scanning may take some time. As we save the result,
we can
avoid rescanning the sources as long as they don't change.

=item Derive a class from ExtUtils::XSBuilder::WrapXS

The WarpXS class is resonsible for takeing the tables which contains the
about the scanned sources and from the map files (see below) and create the
XS code.
As with the ParseSource class, you have to override this call with your own
to tell WrapXS what to do.

See L<ExtUtils::XSBuilder::WarpXS> for a list of overrideable methods.

=item Create map files

XSBuilder will not automaticly create XS functions for all C function and
structure. You
have to give some hints. This is basicly done via the map files. If you
don't change the
default they live under C<xs/maps>. There four map types. For each type
there could
multiple files prefixed with C<foo_>:

=over 4


Contains the mapping from C type to Perl classes


Contains the mapping form C functions to Perl functions. Can be used to
reorder arguments, tell XSBuilder which arguments are actualy return values
and in which Perl package the function will be created.


Contains the mapping from C structures to Perl classes and defines for which
members a access methods should be created. You can also specify if you want
C<new> method for the class.


Contains the mapping form C callback functions to Perl callback functions.
Can be used to
reorder arguments, tell XSBuilder which arguments are actualy return values
and in which Perl package the function will be created.


For a detailed description of the format of the map files see below.

To have a starting point XSBuilder is able to create default map files,
which simply
include all types, functions and structures. You can rerun this map file
everytime and XSBuilder will append all items, that are not already in the
maps files.

If your derived class is called MyClass::WarpXS you simply start the
of the maps files with

    perl -MMyClass::WarpXS -e 'MyClass::WarpXS->checkmaps(" ")'

The the argument to checkmaps give the character that should be at the first
of the new map entries. If you give no argument at all, no map files are
but checkmaps will only compare what is missing. (You need to print the
result somehow
e.g. by using Data::Dumper).
You may also put this into a small script to ease usage and maybe set the
Perl libpath etc.

After you have created your default maps, you have at least to edit the
file, which contains all types that found in the sources. Simply append the
class or the typename
to the line spearated by a C<|> e.g.

    int                 | IV
    struct request_rec  | Apache::RequestRec

=item Create the XS files

Now we can create the code. By starting

    perl -MMyClass::WarpXS -e 'MyClass::WarpXS->run'

XSBuilder will create the XS, pm and Makefile.PL files for every module that
is mentioned in the maps. The result is placed as a directory hierarchy
WrapXS. To control the content of the C<Makefile.PL> and the C<pm> file, you
can override the methods C<makefilepl_text> and C<pm_text>. You can include
addtional code in the XS files, by writing an include file which is included
at the top of the XS file. This file can contain helper functions that can't
automaticly generated. The files must be placed under the C<xs> directory,
with the correct path and name. E.g. to have a header file included for the
module Apache::DAV, create a file C<xs/Apache/DAV/Apache__DAV.h>. The same
can be done for inclusion in the pm file. The name for the above example
would be C<xs/Apache/DAV/DAV_pm>.

=head1 Format of the maps files

For all map files blank lines are ignored and lines starting with a C<#> are
treaded as comments and also ignored.


Contains the mapping from C type to Perl classes.

Format is the name of the C type followed by the name of the Perl class
or the XS type specifier, separated by a |. Example:

    int                 | IV
    struct request_rec  | Apache::RequestRec

If you have a Perl class with a single name namespace (e.g. Apache) you
need to postfix it with two colons (e.g. Apache::). Structures always
needs to be written as "struct foo", also when a typedef for "foo"
Addionaly you can give the id for the typemap if you need a special
conversion and one or more other names for the struct:

    struct request_rec  | Apache::RequestRec | T_APACHEOBJ | r

an optional fivth parameter specifies that the data needs to be copied
when assigned to a struct member and selects the way how memory is

    char *   | PV | | | strdup

The actual code for memory allocation is provided inside the structure map

    MALLOC=strdup:$dest = ($type)ap_pstrdup(obj -> pool, $src)
    MALLOC=malloc:ap_palloc(obj -> pool, $src, sizeof($type)) ;

This gives two ways to allocate memory and copy the data into it. The fives
parameter in the type map selects which of these two should be used.
$src, $dest and $type are replaced by the source, the destionation and the
C<obj> is a pointer to the C-structure.


Contains the mapping form C functions to Perl functions. Can be used to
reorder arguments, tell XSBuilder which arguments are actualy return values
and in which Perl package the function will be created.

There are some keywords which affects all functions follwing that keyword
until a new values for that keyword is given:

=over 4

=item MODULE

the module name (file name) where the function should be placed in
e.g. Apache::Connection -> Apache/Connection.{pm,xs}


the package name functions belong to, defaults to MODULE
value of 'guess' indicates that package name should be
guessed based on first argument found that maps to a Perl class
fallsback on the prefix (ap_ -> Apache, apr_ -> APR)

=item PREFIX

prefix to be stripped
defaults to PACKAGE, converted to C name convention, e.g.
APR::Base64 -> apr_base64_
if the converted prefix does not match, defaults to ap_ or apr_


The format of entries is:

    C function name | dispatch function name (dispatch argspec) | argspec |
Perl alias

Dispatch function name (the C function that is actually called) defaults
to C function name
if the dispatch name is just a prefix (mpxs_, MPXS_)
the C function name is appended to it
the return type can be specified before the C function name,
defaults to return_type in {foo}::FunctionTable as generated by
the ParseSource module.
The dispatch argspec is optional, if given you can use it to pass differnt
parameters to the dispatch function then to the xs function.
If the function name beginns with DEFINE_ a new function is defined,
(for defining function that are not parsed from the source). argsec
must be given also. For the real function name the DEFINE_ is removed.

The argspec defaults to arguments in {foo}::FunctionTable as generated by
the ParseSource module
argument types can be specified to override those in the FunctionTable
default values can be specified, e.g. arg=default_value

  ap_get_client_block   | mpxs_ | r, SV *:buffer, bufsiz
  ap_setup_client_block |       | r, read_policy=REQUEST_CHUNKED_ERROR
  ap_make_array      | ap_make_array(r->pool, nelts, elt_size) | request_rec
*:r, nelts, elt_size

argspec of '...' indicates passthru, calling the function with

    (aTHX_ I32 items, SP **sp, SV **MARK)

To mark an argument as return only you can prefix it with < e.g.

    dav_open_lockdb | | r, ro, <lockdb

will be called as ($error get the return value of the C function)

    ($error, $lockdb) = $r -> open_lockdb (0) ;

The return argument (e.g. lockdb) will always be passed by address
to the function.

the alias will be created in the current PACKAGE

function names that do not begin with /^\w/ or space are skipped.
You can prefix each function name with the following symbols:

    '!' => 'disabled or not yet implemented',
    '~' => 'implemented but not auto-generated',
    '-' => 'likely never be available to Perl',
    '>' => '"private" to apache',
    '?' => 'unclassified',


Contains the mapping from C structures to Perl classes and defines for which
members a access methods should be created. You can also specify if you want
C<new> method for the class.

The format looks like the following:


An optional module name can be given, so the module name specifies in which
file the code should be placed, which the package name is determinated from
the type map file. Example:

 <struct_name MODULE=My::Module>

For all members that are listed here, XSBuilder will generate an access
to read and write it's content. If you want to name the perl access method
different than the C member, you can write

   cMemberValue | member_value

this will map the C<cMemberValue> structure member to the access function
C<member_value>. Default is to use the same name in Perl as in C.
If you give the C<new> member XSBuilder will
create a new method for that class, which can be used to create a new
of that class and initialize it with initial data.


same format as function map, but contains the callbacks


Gerald Richter    ecos electronic communication services gmbh
Internetconnect * Webserver/-design/-datenbanken * Consulting

Post:       Tulpenstrasse 5         D-55276 Dienheim b. Mainz
E-Mail:         Voice:    +49 6133 925131
WWW:      Fax:      +49 6133 925152

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message