incubator-lucy-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marvin Humphrey <>
Subject Inlining XS code
Date Mon, 26 Mar 2007 10:28:26 GMT

If you're not interested in Lucy's Perl bindings, you can skip this  

The most powerful, flexible way to glue C and Perl together is a  
suite of C macros and code-generation tools collectively known as  
"XS".  Another popular option is Inline::C, which allows you to embed  
C code directly within perl code. (Peeking underneath the hood  
reveals that Inline::C is actually an XS code generator).  You can  
also use SWIG, but that's not common.

Ordinarily, XS resides in dedicated .xs files.  There is no one place  
that these files are "supposed" to go -- while Perl modules  
themselves typically end up in $DIST_ROOT/lib, XS files might end up  
in $DIST_ROOT, or in $DIST_ROOT/xs, or alongside their companion Perl  
modules in $DIST_ROOT/lib, or somewhere else.

KinoSearch does something a little bit unconventional with regards to  
its XS code: it's stored Inline::C-style, within the Perl modules  
after an __END__ token, delimited by __XS__ and __POD__ tokens.   
(Example at <>.)  At build time, the XS code is  
extracted and concatenated into a single KinoSearch.xs file, which is  
subsequently compiled using standard Perl building tools.

There are two main reasons for doing things this way.

First, having XS code right next to the perl code it services is a  
real boon.  It's easier to move subroutine implementations from one  
space to the other, as there are 3 files (.c, .h and .pm) to deal  
with instead of 4 (with the addition of the .xs file).  The XS and  
Perl are often tightly coupled; for example, the default values for  
an XS subroutine might be managed via a hash created in Perl-space.   
And even when there is no coupling, it is useful to have all the  
subroutines available from a given perl package concentrated in one  

Second, all that XS code really does belong in one giant file.  It  
would be highly annoying if KS adopted a 1:1 pairing of Perl modules  
to XS files, because XS takes a lllloooonnngggg time to compile.  I  
tried breaking things up into 6 packages a while back but quickly  
reverted because of exploding compilation times.  However, at 4400  
lines and growing, a monolithic KinoSearch.xs file is cumbersome to  

The downside of inlining the XS code is that the style is a bit  
unusual: an XS hacker doesn't automatically know what to make of  
a .pm file like that.  I'm not sure that's a big deal, though,  
because there are so few XS hackers and they're all savvy enough to  
grok what's going on after a short explanation -- learning XS is a  
daunting, difficult task and I'd guess that the number of people who  
know it is in the low thousands, max.    Plus, after a significant  
push last year the XS code in KinoSearch is now 90% glue -- the  
algorithmic stuff is all in either C or Perl.  That will be the case  
with Lucy as well.

Nevertheless, I figured I'd try the more conventional approach of  
using a dedicated XS file for Lucy, so I set up KS to use one as an  
experiment.  I've been unhappy with the results.  The inlining is  
just more development-friendly.

So unless someone here raises an objection, I plan to duplicate that  
style in Lucy's Perl bindings.

Marvin Humphrey
Rectangular Research

View raw message