ant-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bugzi...@apache.org
Subject DO NOT REPLY [Bug 36290] - <copy filtering="on"> mutilates LATIN1 text files
Date Fri, 08 Sep 2006 08:38:14 GMT
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG·
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=36290>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND·
INSERTED IN THE BUG DATABASE.

http://issues.apache.org/bugzilla/show_bug.cgi?id=36290


peterreilly@apache.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED




------- Additional Comments From peterreilly@apache.org  2006-09-08 08:38 -------
I have added a long winded readme to the <copy>
task regarding file encoding:

  <p>
    <strong><a name="encoding">Important Encoding Note:</a></strong>
    The reason that binary files when filtered get corrupted is that
    filtering involves reading in the file using a Reader class. This
    has an encoding specifing how files are encoded. There are a number
    of different types of encoding - UTF-8, UTF-16, Cp1252, ISO-8859-1,
    US-ASCII and (lots) others. On Windows the default character encoding
    is Cp1252, on Unix it is usually UTF-8. For both of these encoding
    there are illegal byte sequences (more in UTF-8 than for Cp1252).
  </p>
  <p>
    How the Reader class deals with these illegal sequences is up to the
    implementation
    of the character decoder. The current Sun Java implemenation is to
    map them to legal characters. Previous Sun Java (1.3 and lower) threw
    a MalformedInputException. IBM Java 1.4 also thows this exception.
    It is the mapping of the characters that cause the corruption.
  </p>
  <p>
    On Unix, where the default is normally UTF-8, this is a <em>big</em>
    problem, as it is easy to edit a file to contain non US Ascii characters
    from ISO-8859-1, for example the Danish oe character. When this is
    copied (with filtering) by Ant, the character get converted to a
    question mark (or some such thing).
  </p>
  <p>
    There is not much that Ant can do. It cannot figure out which
    files are binary - a UTF-8 version of Korean will have lots of
    bytes with the top bit set. It is not informed about illegal
    character sequences by current Sun Java implementions.
  </p>
  <p>
    One trick for filtering containing only US-ASCII is to
    use the ISO-8859-1 encoding. This does not seem to contain
    illegal character sequences, and the lower 7 bits are US-ASCII.
    Another trick is to change the LANG environment variable from
    something like "us.utf8" to "us".
  </p>


-- 
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org


Mime
View raw message