commons-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebb (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Edited] (CODEC-127) Non-ascii characters in source files
Date Mon, 15 Aug 2011 20:09:29 GMT

    [ https://issues.apache.org/jira/browse/CODEC-127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085110#comment-13085110
] 

Sebb edited comment on CODEC-127 at 8/15/11 8:07 PM:
-----------------------------------------------------

I now get:

{code}
commons-codec-generics/src/test/org/apache/commons/codec/language/ColognePhoneticTest.java:110
     {"m├Ânchengladbach", "664645214"},
commons-codec-generics/src/test/org/apache/commons/codec/language/ColognePhoneticTest.java:130
     String[][] data = {{"bergisch-gladbach", "174845214"}, {"M├╝ller-L├╝denscheidt",
"65752682"}};
commons-codec-generics/src/test/org/apache/commons/codec/language/ColognePhoneticTest.java:137
            {"Meyer", "M├╝ller"},
commons-codec-generics/src/test/org/apache/commons/codec/language/ColognePhoneticTest.java:143
            {"ganz", "Gänse"},
commons-codec-generics/src/test/org/apache/commons/codec/language/DoubleMetaphoneTest.java:1222
    this.getDoubleMetaphone().isDoubleMetaphoneEqual("´┐¢", "S");
commons-codec-generics/src/test/org/apache/commons/codec/language/DoubleMetaphoneTest.java:1227
    this.getDoubleMetaphone().isDoubleMetaphoneEqual("´┐¢", "N");
commons-codec-generics/src/test/org/apache/commons/codec/language/bm/BeiderMorseEncoderTest.java:93
String[] names = { "ácz", "átz", "Ignácz", "Ignátz", "Ignác" };
commons-codec-generics/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:47
          { "Nu├▒ez", "spanish", EXACT },
commons-codec-generics/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:49
          { "─îapek", "czech", EXACT },
commons-codec-generics/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:52
          { "Küçük", "turkish", EXACT },
commons-codec-generics/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:55
          { "Ceauşescu", "romanian", EXACT },
commons-codec-generics/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:57
          { "╬æ╬│╬│╬Á╬╗¤î¤Ç╬┐¤à╬╗╬┐¤é", "greek", EXACT
},
commons-codec-generics/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:58
          { "ðƒÐâÐêð║ð©ð¢", "cyrillic", EXACT },
commons-codec-generics/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:59
          { "ÎøÎö΃", "hebrew", EXACT },
commons-codec-generics/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:60
          { "ácz", "any", EXACT },
commons-codec-generics/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:61
          { "átz", "any", EXACT } });
{code}

and

{code}
commons-codec/src/test/org/apache/commons/codec/language/ColognePhoneticTest.java:110    
    {"m├Ânchengladbach", "664645214"},
commons-codec/src/test/org/apache/commons/codec/language/ColognePhoneticTest.java:130    
  String[][] data = {{"bergisch-gladbach", "174845214"}, {"M├╝ller-L├╝denscheidt",
"65752682"}};
commons-codec/src/test/org/apache/commons/codec/language/ColognePhoneticTest.java:137    
     {"Meyer", "M├╝ller"},
commons-codec/src/test/org/apache/commons/codec/language/ColognePhoneticTest.java:143    
     {"ganz", "Gänse"},
commons-codec/src/test/org/apache/commons/codec/language/DoubleMetaphoneTest.java:1227   
  this.getDoubleMetaphone().isDoubleMetaphoneEqual("´┐¢", "S");
commons-codec/src/test/org/apache/commons/codec/language/DoubleMetaphoneTest.java:1232   
  this.getDoubleMetaphone().isDoubleMetaphoneEqual("´┐¢", "N");
commons-codec/src/test/org/apache/commons/codec/language/bm/BeiderMorseEncoderTest.java:93
 String[] names = { "ácz", "átz", "Ignácz", "Ignátz", "Ignác" };
commons-codec/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:47 
         { "Nu├▒ez", "spanish", EXACT },
commons-codec/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:49 
         { "─îapek", "czech", EXACT },
commons-codec/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:52 
         { "Küçük", "turkish", EXACT },
commons-codec/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:55 
         { "Ceauşescu", "romanian", EXACT },
commons-codec/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:57 
         { "╬æ╬│╬│╬Á╬╗¤î¤Ç╬┐¤à╬╗╬┐¤é", "greek", EXACT
},
commons-codec/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:58 
         { "ðƒÐâÐêð║ð©ð¢", "cyrillic", EXACT },
commons-codec/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:59 
         { "ÎøÎö΃", "hebrew", EXACT },
commons-codec/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:60 
         { "ácz", "any", EXACT },
commons-codec/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:61 
         { "átz", "any", EXACT } });
{code}

This was using an updated version of the script that uses File::Find to process directory
traversal better.
(Some lines shortened above by manually removing leading spaces)

I think all the actual errors have now been fixed.

The remaining lines contain some non-ASCII characters which could be replaced by Unicode escapes
for better portability.
However, that would make it harder to read the code in some cases.
So I'm thinking of using Unicode escapes in the Strings, but adding the original as an end-of-line
comment.
The comments might still get mangled, but at least the code would not, and it would be easy
to reconstruct the comments from the Unicode.

WDYT?

      was (Author: sebb@apache.org):
    What error do you get? Just curious.

I now get:

{code}
commons-codec-generics/src/test/org/apache/commons/codec/language/ColognePhoneticTest.java:110
     {"m├Ânchengladbach", "664645214"},
commons-codec-generics/src/test/org/apache/commons/codec/language/ColognePhoneticTest.java:130
     String[][] data = {{"bergisch-gladbach", "174845214"}, {"M├╝ller-L├╝denscheidt",
"65752682"}};
commons-codec-generics/src/test/org/apache/commons/codec/language/ColognePhoneticTest.java:137
            {"Meyer", "M├╝ller"},
commons-codec-generics/src/test/org/apache/commons/codec/language/ColognePhoneticTest.java:143
            {"ganz", "Gänse"},
commons-codec-generics/src/test/org/apache/commons/codec/language/DoubleMetaphoneTest.java:1222
    this.getDoubleMetaphone().isDoubleMetaphoneEqual("´┐¢", "S");
commons-codec-generics/src/test/org/apache/commons/codec/language/DoubleMetaphoneTest.java:1227
    this.getDoubleMetaphone().isDoubleMetaphoneEqual("´┐¢", "N");
commons-codec-generics/src/test/org/apache/commons/codec/language/bm/BeiderMorseEncoderTest.java:93
String[] names = { "ácz", "átz", "Ignácz", "Ignátz", "Ignác" };
commons-codec-generics/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:47
          { "Nu├▒ez", "spanish", EXACT },
commons-codec-generics/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:49
          { "─îapek", "czech", EXACT },
commons-codec-generics/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:52
          { "Küçük", "turkish", EXACT },
commons-codec-generics/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:55
          { "Ceauşescu", "romanian", EXACT },
commons-codec-generics/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:57
          { "╬æ╬│╬│╬Á╬╗¤î¤Ç╬┐¤à╬╗╬┐¤é", "greek", EXACT
},
commons-codec-generics/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:58
          { "ðƒÐâÐêð║ð©ð¢", "cyrillic", EXACT },
commons-codec-generics/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:59
          { "ÎøÎö΃", "hebrew", EXACT },
commons-codec-generics/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:60
          { "ácz", "any", EXACT },
commons-codec-generics/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:61
          { "átz", "any", EXACT } });
{code}

and

{code}
commons-codec/src/test/org/apache/commons/codec/language/ColognePhoneticTest.java:110    
    {"m├Ânchengladbach", "664645214"},
commons-codec/src/test/org/apache/commons/codec/language/ColognePhoneticTest.java:130    
  String[][] data = {{"bergisch-gladbach", "174845214"}, {"M├╝ller-L├╝denscheidt",
"65752682"}};
commons-codec/src/test/org/apache/commons/codec/language/ColognePhoneticTest.java:137    
     {"Meyer", "M├╝ller"},
commons-codec/src/test/org/apache/commons/codec/language/ColognePhoneticTest.java:143    
     {"ganz", "Gänse"},
commons-codec/src/test/org/apache/commons/codec/language/DoubleMetaphoneTest.java:1227   
  this.getDoubleMetaphone().isDoubleMetaphoneEqual("´┐¢", "S");
commons-codec/src/test/org/apache/commons/codec/language/DoubleMetaphoneTest.java:1232   
  this.getDoubleMetaphone().isDoubleMetaphoneEqual("´┐¢", "N");
commons-codec/src/test/org/apache/commons/codec/language/bm/BeiderMorseEncoderTest.java:93
 String[] names = { "ácz", "átz", "Ignácz", "Ignátz", "Ignác" };
commons-codec/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:47 
         { "Nu├▒ez", "spanish", EXACT },
commons-codec/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:49 
         { "─îapek", "czech", EXACT },
commons-codec/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:52 
         { "Küçük", "turkish", EXACT },
commons-codec/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:55 
         { "Ceauşescu", "romanian", EXACT },
commons-codec/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:57 
         { "╬æ╬│╬│╬Á╬╗¤î¤Ç╬┐¤à╬╗╬┐¤é", "greek", EXACT
},
commons-codec/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:58 
         { "ðƒÐâÐêð║ð©ð¢", "cyrillic", EXACT },
commons-codec/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:59 
         { "ÎøÎö΃", "hebrew", EXACT },
commons-codec/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:60 
         { "ácz", "any", EXACT },
commons-codec/src/test/org/apache/commons/codec/language/bm/LanguageGuessingTest.java:61 
         { "átz", "any", EXACT } });
{code}

This was using an updated version of the script that uses File::Find to process directory
traversal better.
(Some lines shortened above by manually removing leading spaces)

I think all the actual errors have now been fixed.

The remaining lines contain some non-ASCII characters which could be replaced by Unicode escapes
for better portability.
However, that would make it harder to read the code in some cases.
So I'm thinking of using Unicode escapes in the Strings, but adding the original as an end-of-line
comment.
The comments might still get mangled, but at least the code would not, and it would be easy
to reconstruct the comments from the Unicode.

WDYT?
  
> Non-ascii characters in source files
> ------------------------------------
>
>                 Key: CODEC-127
>                 URL: https://issues.apache.org/jira/browse/CODEC-127
>             Project: Commons Codec
>          Issue Type: Bug
>            Reporter: Sebb
>
> Some of the test cases include characters in a native encoding (possibly UTF-8), rather
than using Unicode escapes.
> This can cause a problem for IDEs if they don't know the encoding (e.g. cause compilation
errors, which is how I found the issue), and possibly some transformations may corrupt the
contents, e.g. fixing EOL.
> I think we should have a rule of using Unicode escapes for all such non-ascii characters.
> It's particularly important for non-ISO-8859-1 characters.
> Some example classes with non-ascii characters:
> {code}
> binary\Base64Test.java:96         byte[] decode = b64.decode("SGVsbG{´┐¢´┐¢´┐¢´┐¢´┐¢´┐¢}8gV29ybGQ=");
> language\ColognePhoneticTest.java:110             {"m├Ânchengladbach", "664645214"},
> language\ColognePhoneticTest.java:130         String[][] data = {{"bergisch-gladbach",
"174845214"}, {"M├╝ller-L├╝denscheidt", "65752682"}};
> language\ColognePhoneticTest.java:137             {"Meyer", "M├╝ller"},
> language\ColognePhoneticTest.java:143             {"ganz", "Gänse"},
> language\DoubleMetaphoneTest.java:1222         this.getDoubleMetaphone().isDoubleMetaphoneEqual("´┐¢",
"S");
> language\DoubleMetaphoneTest.java:1227         this.getDoubleMetaphone().isDoubleMetaphoneEqual("´┐¢",
"N");
> language\SoundexTest.java:367         if (Character.isLetter('´┐¢')) {
> language\SoundexTest.java:369                 Assert.assertEquals("´┐¢000", this.getSoundexEncoder().encode("´┐¢"));
> language\SoundexTest.java:375             Assert.assertEquals("", this.getSoundexEncoder().encode("´┐¢"));
> language\SoundexTest.java:387         if (Character.isLetter('´┐¢')) {
> language\SoundexTest.java:389                 Assert.assertEquals("´┐¢000", this.getSoundexEncoder().encode("´┐¢"));
> language\SoundexTest.java:395             Assert.assertEquals("", this.getSoundexEncoder().encode("´┐¢"));
> {code}
> The characters are probably not correct above, because I used a crude perl script to
find them:
> {code}
> perl -ne "$.=1 if $s ne $ARGV;print qq($ARGV:$. $_) if m/\P{ASCII}/;$s=$ARGV;" xxxx.java
> {code}
> language\SoundexTest.java:367 in particular is incorrect, because it's supposed to be
a single character.
> Now one might think that native2ascii -encoding UTF-8 would fix that, but it gives:
> if (Character.isLetter('\ufffd'))
> which is an "unknown" character.
> Similarly for binary\Base64Test.java:96.
> It's not all that clear what the Unicode escapes should be in these cases, but probably
not the unknown character.
> [Possibly the characters got mangled at some point, or maybe they have always been wrong]
> The ColognePhoneticTest.java cases are less serious, as the characters are valid ISO-8859-1
(accented German), but given that the rest of the file uses unicode escaps, I think they should
be changed too (but add comments to say what they are, e.g. o-umlaut, u-umlaut)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message