Mailing-List: contact lucy-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: lucy-user@incubator.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Date: Wed, 12 Oct 2011 10:47:02 -0700
From: Marvin Humphrey <marvin@rectangular.com>
To: lucy-user@incubator.apache.org
Cc: peter@peknet.com
Message-ID: <20111012174702.GA9586@rectangular.com>
References: 
 <CAKn6EF88=Fb8_tP7x3wHs1nV49YEHTUQPBPwGRNYDzV69KMg6Q@mail.gmail.com>
 <4E95A611.3050100@peknet.com>
 <CAKn6EF__HxEzYDrHFwHMi907L2W9O_CU8OnX9dX4F1YMa-ryEQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
 <CAKn6EF__HxEzYDrHFwHMi907L2W9O_CU8OnX9dX4F1YMa-ryEQ@mail.gmail.com>
User-Agent: Mutt/1.5.18 (2008-05-17)
Subject: Re: [lucy-user] Different UTF-8 behaviour between perl 5.8.8
	(indexes ok) and 5.10.1 (indexing fails)

On Wed, Oct 12, 2011 at 05:57:33PM +0200, goran kent wrote:
> > It sounds like, without seeing a reproduce-able test case, that Lucy is
> > choking appropriately on malformed UTF-8.
> 
> Absolutely.  What's interesting is that the same Lucy code does not
> choke on the other machines with the older Perl.

Lucy trusts that incoming data it has received from Perl is well-formed.
(Technically, it assumes that string data obtained via the XS routine
SvPVutf8() is well-formed UTF-8, notwithstanding the difference between Perl's
loose internal representation and the Unicode standard for UTF-8.)  We could
add an index-time validity check, but that would slow down indexing.

At search-time, though, Lucy is reading from the file system rather than
receiving data from Perl -- and data from the file system cannot be trusted.
Therefore, Lucy always performs validity checks when reading what is
ostensibly UTF-8 data out of an existing index.

I don't know of a mechanism whereby Lucy's behavior would change between
different versions of Perl.  In any case, having invalid UTF-8 in your Perl
scalars is bad news -- it can do things like crash the regex engine.  It will
also lead to corrupt Lucy indexes that fail the search-time UTF-8 validity
check.

How are you getting raw data into Perl?

> Anyway, I like the idea of rolling my own perl to be absolutely sure
> of coherence across my machines.

+1

Marvin Humphrey