commons-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benedikt Ritter (JIRA)" <>
Subject [jira] [Commented] (LANG-977) NumericEntityEscaper incorrectly encodes supplementary characters
Date Sat, 15 Feb 2014 15:44:19 GMT


Benedikt Ritter commented on LANG-977:

Hello Chris,

We don't maintain a fixed release schedule or something like that. I planned to roll out 3.3
in early February. Currently I don't really have the time, but I'm still planning to release
3.3 in February.


> NumericEntityEscaper incorrectly encodes supplementary characters
> -----------------------------------------------------------------
>                 Key: LANG-977
>                 URL:
>             Project: Commons Lang
>          Issue Type: Bug
>          Components: lang.text.translate.*
>    Affects Versions: 3.2.1
>            Reporter: Chris Karcher
>            Assignee: Benedikt Ritter
>             Fix For: 3.3
>         Attachments: NumericEntityEscaper.patch
> NumericEntityEscaper will incorrectly encode supplementary unicode characters depending
on the char length of the first code point in the string.
> To reproduce, run:
> {code}
> String escaped = NumericEntityEscaper.between(0x7f, Integer.MAX_VALUE).translate("a \uD83D\uDC14
> {code}
> Expected:
> {code}
> escaped == "a &#128020; &#128169;"
> {code}
> Actual:
> {code}
> escaped == "a &#128020;&#56340; &#128169;&#56489;"
> {code}
> The issue lies in CharSequenceTranslator.translate() and the way it checks code points
to figure out how many characters it needs to consume.  Specifically, the issue is on [line
> {code}
> // contract with translators is that they have to understand codepoints 
> // and they just took care of a surrogate pair
> for (int pt = 0; pt < consumed; pt++) {
>     pos += Character.charCount(Character.codePointAt(input, pt));
> }
> {code}
> The point of this code is to check the charCount of the character that was just translated
and move ahead by that many characters in the input string.  The bug is that it's indexing
into the string using 'pt', which is _always_ 0 at the beginning of the loop.  It's effetively
checking the charCount of first character in the string every time.
> A patch is attached that fixes the issue and includes supporting unit tests.  Fixing
this issue in CharSequenceTranslator uncovered an issue in CsvEscaper/CsvUnescaper caused
by the fact that it wasn't respecting the "code point contract" described in CharSequenceTranslator.translate.
 The fix there was to have the translate methods return the string's code point count rather
than character count.

This message was sent by Atlassian JIRA

View raw message