lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] Commented: (LUCENE-2016) replace invalid U+FFFF character during indexing
Date Thu, 29 Oct 2009 18:07:59 GMT


Michael McCandless commented on LUCENE-2016:

bq. Finally, I completely disagree with the nontrivial performance comment. The trick is to
make sure the execution branch / checks for the process-internal characters outside the bmp,
only occurs for surrogate pairs. They are statistically very rare and if done right, it will
not affect performance of BMP content.

OK I agree, you're right: we could in fact do this with negligible impact to performance.

bq. Its my understanding Lucene indexes should be portable to different programming languages:
perhaps my implementation in C/perl/python decides to use a different process-internal character,
this is allowed by Unicode and I think we should adhere to it, I don't think its being anal.

But if we forcefully map all invalid-for-interchange unicode characters to the replacement
character (I think that's what's being proposed, right?), then your app no longer has any
characters it can use for its own "internal" purposes?

Can you open a new issue to track this?  This is a wider discussion than preventing index
corruption :)

> replace invalid U+FFFF character during indexing
> ------------------------------------------------
>                 Key: LUCENE-2016
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 2.4, 2.4.1, 2.9
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 2.9.1, 3.0
>         Attachments: LUCENE-2016.patch
> If the invalid U+FFFF character is embedded in a token, it actually causes indexing to
silently corrupt the index by writing duplicate terms into the terms dict.  CheckIndex will
catch the error, and merging will hit exceptions (I think).
> We already replace invalid surrogate pairs with the replacement character U+FFFD, so
I'll just do the same with U+FFFF.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message