lucy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marvin Humphrey <mar...@rectangular.com>
Subject Re: [lucy-user] SearchServer / ClusterSearcher - massive performance hit
Date Fri, 26 Oct 2012 03:03:20 GMT
On Thu, Oct 25, 2012 at 12:56 PM, Dag Lem <dag@nimrod.no> wrote:
> I don't have any experience with the many solutions for network
> communication. However if, for the sake of argument, someone
> implemented a solution using libev (using the Perl EV module?), why
> would it be problematic to include it as a dependency for Lucy, as
> Marvin stated?

The first issue is licensing compatibility with Lucy's permissive Apache
License 2.0.

The libev C library is under a BSD license, which is compatible with the ALv2.
However, the Perl EV bindings are under "the same terms as Perl itself" --
GPL/Artistic-1 -- which ASF legal has not approved for use by Apache projects.

So we can use the libev C library but not its Perl bindings.

Incidentally, this is exactly the same licensing situation we were in with the
Snowball stemming library.  Our solution was to ditch the Perl bindings and
bundle the C library; the same solution is available to us with libev.

The second issue is that since Lucy is intended to be a low-level library, we
tend not to add dependencies unless there's a really strong justification and
they meet stringent criteria for portability and reliability.  All our
existing dependencies (Snowball, utf8proc and Lemon) are bulletproof -- they
build everywhere with no problems.

Judging from CPAN testers, libev isn't bulletproof, just good.  It looks like
it has the same problem as Lucy: while it aspires to be portable to all common
modern operating systems, its lofty ambitions for functionality make it hard
to meet that goal all the time:

    www.cpantesters.org/distro/E/EV.html

    http://www.cpantesters.org/cpan/report/808f4c72-b016-11e1-8953-99ddd5b1ea5c

    In file included from EV.xs:35:
    libev/ev.c:941:3: #error "memory fences not defined for your
architecture, please report"
    *** Error code 1

    http://www.cpantesters.org/cpan/report/11772f04-cdc0-11e1-ac3c-b54b049ce4cd

    dyld: /var/root/perl5/perlbrew/perls/perl-5.14.1/bin/perl Undefined symbols:
/private/var/root/.cpan/build/EV-4.11-HnG02y/blib/arch/auto/EV/EV.bundle
undefined reference to _pthread_atfork expected to be defined in a
dynamic image


I'm not fundamentally opposed to adding libev as a bundled dependency -- at
least it's nice and small -- but since adding it would degrade Lucy's
portability, we should make sure that we really need it.

> In any case, I believe it's a good idea to first sort out any high(er)
> level issues which can impede performance on a more fundamental level
> (like network roundtrips).

+1

The stuff we're working on now will benefit us no matter what direction we
take.

> I don't have any Lucy-fu (nor much time, unfortunately) to take on any
> of the fundamental issues.

Well, the more you give, the more you get.  :)

Marvin Humphrey

Mime
View raw message