lucy-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marvin Humphrey <>
Subject Re: [lucy-dev] Unicode integration
Date Wed, 16 Nov 2011 03:49:32 GMT
On Tue, Nov 15, 2011 at 09:51:49PM +0100, Nick Wellnhofer wrote:
> One alternative I could find is utf8proc [2]. It's 20K of C code,  
> MIT-licensed and used for Postgres extensions and a Ruby gem. It  
> supports Unicode normalization, case folding and stripping of accents.

utf8proc also supplies some low-level routines which we might be able to use
to replace stuff in Lucy/Util/StringHelper.c.

It compiles plenty fast, it's just two .c files (one of which includes the
other) and one .h file.  The Makefile isn't portable, but I see that they've
taken pains to accommodate MSVC in the C files, so they'll probably compile
everywhere that Lucy does.

I see no problems with bundling utf8proc as a dependency.

Looks like a great find, Nick!

> One downside of bundling a Unicode library is that they all need some  
> rather large tables. utf8proc comes with a 1.2 MB .c file containing the  
> tables. The whole library compiles to about 500 KB. Unicode::Normalize  
> builds its tables from the Unicode database files that come with Perl  
> and compiles to about 300 KB. All this on i386, 32 bit.

With the current state of things, adding 500 KB is unlikely to make a

On my Mac running Snow Leopard, adding utf8proc pushes the compiled size of
Lucy.bundle from 2.8 MB to 3.3 MB.  The largest compiled objects contributing
to that tally are Lucy.o at 1.2 MB, compiled from Lucy.xs, and autogen/parcel.c
at 1.2 MB, which contains Clownfish OO support such as vtables.  The Snowball
stemmers add around 200 KB, and the Snowball stoplists around 100 KB.

If we put our minds to it, we could slim down Lucy.o and parcel.o -- maybe by
a lot, since nobody's ever bothered to work on optimizing for space.  The same
wouldn't hold true for utf8proc (or the Snowball materials).  But that just
doesn't seem important given all the other reasons utf8proc looks like a good

> On the positive side, we'd have things like case folding, normalization  
> and accent stripping directly in core.

It would be great to support accent stripping in Lucy -- that's something a
lot of people need.  Normalization would also be a nice feature to offer
(Maybe we should make it the first step of PolyAnalyzer or PolyAnalyzer's

It would also be great to migrate Lucy::Analysis::CaseFolder code away from
its dependency on the Perl C API.

> We'd also get Unicode features  for new host languages out of the box and
> it's the only way to make sure  Unicode is handled consistently across
> different host languages and  client platforms. The latter might be a rather
> academic concern, though.

Personally, I don't see cross-host index compatibility as so important that we
ought to make big sacrifices to achieve it.

Regardless, integrating utf8proc seems worthwhile for lots of other reasons. 

+1 from me!

Marvin Humphrey

View raw message