Mailing-List: contact dev-help@subversion.apache.org; run by ezmlm
Precedence: bulk
Received-SPF: neutral (nike.apache.org: local policy)
References: <1048611412321192@web20g.yandex.ru>
 <20141003113501.GF1256@ted.stsp.name>   
Message-ID: <1412338519.68585.YahooMailNeo@web87701.mail.ir2.yahoo.com>
Date: Fri, 3 Oct 2014 13:15:19 +0100
From: Julian Foad <julianfoad@btopenworld.com>
Reply-To: Julian Foad <julianfoad@btopenworld.com>
Subject: Re: Subversion binary file detection is look like broken
To: Stefan Sperling <stsp@elego.de>, Navrotskiy Artem <bozaro@ya.ru>
Cc: "dev@subversion.apache.org" <dev@subversion.apache.org>
In-Reply-To: <20141003113501.GF1256@ted.stsp.name>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Stefan Sperling wrote:=0A> On Fri, Oct 03, 2014 at 11:26:32AM +0400, Navrot=
skiy Artem wrote:=0A>>=A0 =A0 Subversion console client try to detect binar=
y file with algorythm:=0A>> =0A>>=A0 =A0=A0=A01. File is NOT BINARY if it c=
ontains only BOM UTF-8 signature (why not=0A>>=A0 =A0 =A0 =A0 check as firs=
t N bytes is corret UTF-8?);=0A>>=A0 =A0=A0=A02. File is BINARY if first 10=
24 bytes contains ZERO byte (uniform=0A>>=A0 =A0 =A0 =A0 distribution of by=
tes takes change of absent ZERO byte: (1 - 1 /=0A>>=A0 =A0 =A0 =A0 256) ^ 1=
024 =3D ~1.8%);=0A>>=A0 =A0=A0=A03. File is BINARY if first 1024 bytes cont=
ains over 85% of characters=0A>>=A0 =A0 =A0 =A0 not in range 0x07-0x0D, 0x2=
0-0x7F (total we have 153 "binary"=0A>>=A0 =A0 =A0 =A0 bytes, ~60%).=0A>> =
=0A>>=A0 =A0 This algoritm looks like broken.=0A=0AThe requirement (3) for =
>85% non-ASCII* bytes =3D> binary, was a historical accident. The =0Aorigin=
al intention was >15% non-ASCII bytes =3D> binary, or in other words >85% A=
SCII bytes =3D> text. Quoting from libsvn_subr/io.c:svn_io_is_binary_data()=
:=0A=0A=A0=A0=A0=A0 NOTE:=A0 Originally, I intended to target 85% of the by=
tes being in=0A=A0=A0=A0=A0 the specified ranges, but I flubbed the conditi=
on.=A0 At any rate,=0A=A0=A0=A0=A0 folks aren't complaining, so I'm not sur=
e that it's worth=0A=A0=A0=A0=A0 adjusting this retroactively now.=0A=0APer=
haps now is the time to change that to match the original intent.=0A=0A* I =
use the term ASCII loosely to mean "bytes in those two ranges".=0A=0A=0A> C=
an you suggest a better algoritm?=0A> =0A>> For example:=0A>>=A0 =A0=A0 1. =
File "text.txt":=0A>> Is file contains text block from wikipedia about subv=
ersion in UTF-8=0A>> (https://ru.wikipedia.org/wiki/Subversion) and=A0unfor=
tunaly contains too=0A>> many cyrillic charactes (on character - 2 "binary"=
 bytes).=0A>>=A0 =A0=A0 2. File "binary.txt" detected as "text"=0A>> It was=
 created by "dd if=3D/dev/urandom of=3Dbinary.txt count=3D1 bs=3D2048" and=
=0A>> unfortunaly does not contains ZERO byte in first 1024 bytes.=0A=0ACha=
nging the 85% condition would fix example 2. However it would make example =
1 occur more often, unless we also make valid UTF-8 be detected as text.=0A=
=0AIt does sound like a good idea to make valid UTF-8 be detected as text.=
=0A=0A- Julian