incubator-lucy-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marvin Humphrey <>
Subject Re: Missing XSUB functions
Date Wed, 18 Mar 2009 22:59:59 GMT
I'm cc-ing this reply to lucy-dev at lucene dot apache dot org.  Boilerplater
is under active discussion there, and it's the best place for this
conversation to continue.

On Wed, Mar 18, 2009 at 09:42:04PM +0100, Andreas Altergott wrote:

> I'm used to have a .xs file containing an entry like
> int
> foo(bar)
>     int bar
> But there are no .xs files in KinoSearch.  

Run these commands:

  perl Build.PL
  ./Build code

KinoSearch.xs will appear.  It's dynamically generated.

Alternatively, run "./Build boilerplater", which is the minimum command to
generate KinoSearch.xs.  ("./Build code" does a bunch more after that, such as
compiling all ".c" files.)

> There are though __XS__ and
> __AUTO_XS__ areas inside the module files.  But again there's nothing
> mentioned about foo(bar).

I used to hand-code all of KinoSearch's XS bindings, but that was tiresome and
error-prone.  The autogenerated KinoSearch.xs file is now over 30,000 lines
long -- I'd really hate to have to maintain that manually.  :)

Any code that follows an __XS__ token in a Perl module is extracted and copied
verbatim into KinoSearch.xs.  All of the code in those blocks is pure,
ordinary, hand-rolled XS code.  The only reason that it lives in the Perl
module files is so that it's easy to find and edit.

(Actually, I just looked and there are only about 1300 lines of hand-coded XS
left.  We might want to extract that from the Perl modules and put it in a
place where people like you would expect to find it.)

The __AUTO_XS__ blocks are small Perl snippets used to build up a set of
arguments to Boilerplater::Binding::Perl.  The only stuff that's left as
hand-coded __XS__ is binding code that *can't* be auto-generated for one
reason or another; __AUTO_XS__ is where most of action is.

However, Boilerplater::Binding::Perl at present has no API documentation, only
comments.  And it's got a lousy interface.  I didn't expect to be explaining
it to anyone this soon.  :)

> So how does Perl now, that if I call KinoSearch::Doc->new it should call
>  Doc* Doc_new(void *fields, i32_t doc_num, float boost)?

This section of the __AUTO_XS__ block in lib/KinoSearch/ tells
Boilerplater::Binding::Perl how to generate the binding for KinoSearch::Doc:

    {   "KinoSearch::Doc" => {
            make_constructors => ['new'],  # <------- constructor spec
            bind_methods => [
                qw( Set_Doc_Num
                    Set_Fields )
            make_pod => {
                methods     => [qw( set_boost get_boost get_fields )],
                synopsis    => $synopsis,
                constructor => { sample => $constructor },

I've included the generated code for the constructor below my sig.  It's a
little long because it does a lot: it handles subclassing, parses labeled
params, performs argument error checking, applies default values, etc.

> I assume it has something to do with the boilerplater which also does a
> lot of dark Voodoo :-D

We can think of Boilerplater as having two parts.

  1) The part that processes the ".bp" files in core, auto-generating a bunch
     of host-language agnostic ".h" files.
  2) The part that auto-generates Perl-specific binding code using XS.

We're concerned about the second half.  That code lives in six perl modules
under trunk/boilerplater/lib/Boilerplater/Binding/Perl* and is analogous to
SWIG, another automatic binding generator which spits out XS binding code based
on function signatures.

> Would the test file 200-doc be a binding or a core test?

Mmm, you picked a real doozy right off the bat. :(  Doc and its subclass
HitDoc are the only classes in all of KS which cannot presently have
host-language agnostic C constructors, because Doc uses a
host-language-specific hash table for storing fields.

So... it ought to be a core test, but it can't be.  It has to be a binding
test, run from Perl in this case.


Marvin Humphrey

# From lib/KinoSearch/ #

%KinoSearch::Doc::new_PARAMS = (
    fields => undef,
    doc_num => 0,
    boost => 1.0,

/************************ From KinoSearch.xs: *********************/

XS(XS_KinoSearch__Doc__new); /* -Wmissing-prototypes */
    if (items < 1)
        CONFESS("Usage: %s(class_name, ...)",  GvNAME(CvGV(cv)));
    SP -= items;

        kino_Doc* self; 
        void* fields;
        chy_i32_t doc_num;
        float boost;
        kino_Doc* retval;
        SV* fields_sv = NULL; 
        SV* doc_num_sv = NULL; 
        SV* boost_sv = NULL; 
        allot_params( &(ST(0)), 1, items, "KinoSearch::Doc::new_PARAMS",
            &fields_sv, "fields", 6,
            &doc_num_sv, "doc_num", 7,
            &boost_sv, "boost", 5,

        if ( fields_sv != NULL && SvOK(fields_sv) ) {
            if (SvROK(fields_sv)) {
                fields = SvRV(fields_sv);
            else {
                fields = NULL; /* avoid uninitialized compiler warning */
                CONFESS("fields is not a reference");
        else {
            fields = NULL; 
        if ( doc_num_sv != NULL && SvOK(doc_num_sv) ) {
            doc_num = (chy_i32_t)SvIV( doc_num_sv );
        else {
            doc_num = 0;
        if ( boost_sv != NULL && SvOK(boost_sv) ) {
            boost = (float)SvNV( boost_sv );
        else {
            boost = 1.0;
        self = (kino_Doc*)new_blank_obj( ST(0) );

        retval = kino_Doc_init(self, fields, doc_num, boost);
        ST(0) = Kino_Obj_To_Native(retval);
        sv_2mortal( ST(0) );


View raw message