lucy-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marvin Humphrey <>
Subject Re: [lucy-dev] OFFSET globals
Date Tue, 24 Apr 2012 11:32:40 GMT
On Mon, Apr 23, 2012 at 10:20 PM, Nathan Kurz <> wrote:
> A fine reason to finally do this right, and having him excited to
> solve this is another.  But I haven't been sure whether it's Nick
> demanding an upgrade proof ABI, or whether you feel it should be one.
> I was wondering whether it would be better to have him continue
> writing extensions and recompiling as necessary until the ABI settles
> down, since the process of writing these extensions will probably
> cause faster change than will be the usual case.
> But as long as _both_ of you view this as a preliminary requirement,
> it's a great thing to get out of the way.

Supporting CPAN/Rubygems/PyPI-style development for compiled extensions is of
paramount importance, IMO.  Extension authors need to be able to rapid
prototype in their host language of choice, and then at the time they deem
appropriate, port their extension to a compiled format for efficiency.  It
cannot be the case that in order to attain maximum performance, features must
be accepted into core, distributed and compiled with core, and then supported
in perpetuity by the core development team.  That development model does not

Of course a feature might go into core because there is consensus that it
belongs there, but it can't be the case that "core is fast and extensions are
slow", creating an unnatural pressure to bloat core that is unhealthy for the

> 1) We want to make it possible to add methods to core Lucy objects on
> a library upgrade without requiring a recompile of extension classes
> that inherit from these.


> 2) We don't want to allow "monkey-patching" of classes at runtime,
> such that an extension can make changes to parent classes that
> propagate back to the child classes.

I think you can stop at "make changes to parent classes [period]".  As far as
Clownfish is concerned, once a class has been registered, it is immutable.

For an extension which is implemented in a host language, you might get away
with some lazy code generation funny business, but that's an implementation
detail and you're on your own.

In particular: once registered, VTables are immutable[1].  No adding new
methods, no modification of method pointers or any member variables.  VTables
are shared globals; non-constant shared globals suck for many, many reasons,
including thread-safety, vulnerability to action-at-a-distance bugs,
resistance to compiler optimizations, etc.

For what it's worth, there is a direct parallel between Lucy allowing addition
of new segments but forbidding modification of existing segment data (enabling
mmap with MMAP_SHARED), and Clownfish allowing incremental compilation of new
classes but forbidding modification of existing ones (allowing VTables to be
shared across threads).

> 3) We want to handle the case of a method that moves from a parent
> class to a grandparent, and vice versa.   As long as it's in the chain
> of ancestors, the exact location shouldn't matter.

+1, well articulated.

> 4) In all other cases (removal or signature change of an inherited
> method) we just want to detect the incompatibility at start time and
> die with an appropriate error message rather than randomly crashing or
> doing the wrong thing.

+0.  Nice if possible, but from the perspective of an existing compiled
extension, it's hard to detect when upstream has broken an ABI.

> 5) It's OK to start with a Linux-only solution, so long as there is a
> pathway to a solution on Windows and OSX, and

+1, so long as there aren't regressions at release points.  It wouldn't be
cool to release version Lucy 0.4.0 and have it broken everywhere but Linux,
but that policy sounds fine for new features.

> it's OK to to have a
> solution that is efficient only on x64 as long as there is an x86
> workaround. (?)


> Are these correct?  Are the requirements I'm missing?

I would add this:

6) It should be possible to remove methods from upstream which are not marked
as "public.

Then, after we finish dealing with methods, the next step is to give upstream
the ability to add, remove or rearrange member variables. :)

Marvin Humphrey

[1] In the present implementation, the cached host object in a VTable is
    mutable.  This is a bug.

View raw message