Mailing-List: contact users-help@subversion.apache.org; run by ezmlm
Precedence: bulk
Received-SPF: pass (athena.apache.org: local policy)
Date: Fri, 11 Oct 2013 19:25:19 +0200
From: Stefan Sperling <stsp@elego.de>
To: Ben Reser <ben@reser.org>
Cc: Branko =?utf-8?B?xIxpYmVq?= <brane@wandisco.com>,
        "users@subversion.apache.org" <users@subversion.apache.org>
Subject: Re: SVN Blame Returns Corrupt Data
Message-ID: <20131011172519.GE24900@ted.stsp.name>
Mail-Followup-To: Ben Reser <ben@reser.org>,
	Branko =?utf-8?B?xIxpYmVq?= <brane@wandisco.com>,
	"users@subversion.apache.org" <users@subversion.apache.org>
References: 
 <CA+g5=ZBym0QYuqxg487gJxVD0qL3+HFqmsE=Qmmydw-YbctVMw@mail.gmail.com>
 <EA9A16E4E64F694BAD262610B0A2515C236B5DAC@USALWEXMBX2.infor.com>
 <52580763.7050608@wandisco.com>
 <EA9A16E4E64F694BAD262610B0A2515C236B5E64@USALWEXMBX2.infor.com>
 <52581247.9040707@wandisco.com>
 <EA9A16E4E64F694BAD262610B0A2515C236B5E93@USALWEXMBX2.infor.com>
 <52581CA2.9080401@wandisco.com>
 <EA9A16E4E64F694BAD262610B0A2515C236B66D8@USALWEXMBX2.infor.com>
 <525825C9.3020104@wandisco.com>
 <52582CCF.2020802@reser.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <52582CCF.2020802@reser.org>
User-Agent: Mutt/1.5.21 (2010-09-15)

On Fri, Oct 11, 2013 at 09:52:31AM -0700, Ben Reser wrote:
> On 10/11/13 9:22 AM, Branko Čibej wrote:
> > You'd have to extend Subversion's file type detection to detect UTF-16.
> > See svn_io_detect_mimetype2 in line 3333 in this file:
> > 
> > http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_subr/io.c?view=markup
> > Subversion currently only looks at the first 1k Bytes of a file. It may
> > be enough to check that this initial part of the file contains only
> > valid UTF-16 (BE or LE) codes.
> 
> Even if all we looked for is the BOM it might be helpful enough.  I suspect the
> development tools producing UTF-16 are including BOMs.  Windows seems to be
> fond of including them, Notepad puts one even on UTF-8.

Couldn't Subversion automatically convert UTF-16 files to UTF-8 before
processing them for diff/merge/blame, and convert output written to
the original files back to UTF-16?

That would require some work because existing streams, strings, and files
passed around in the code would need to be wrapped so that translation
to/from the internal from/to the external encoding is seamless.

But I don't see why such an approach couldn't be made to work in principle.
It might even result in some spring cleaning in the code base and pave the
way for improved handling of file formats such as XML for diff and merge.

What do you think? Is it worth adding this to our project ideas page?