From dev-return-37409-archive-asf-public=cust-asf.ponee.io@subversion.apache.org Fri Feb 2 06:40:39 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 73FA6180608 for ; Fri, 2 Feb 2018 06:40:39 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 62989160C57; Fri, 2 Feb 2018 05:40:39 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 84FC0160C44 for ; Fri, 2 Feb 2018 06:40:38 +0100 (CET) Received: (qmail 86072 invoked by uid 500); 2 Feb 2018 05:40:37 -0000 Mailing-List: contact dev-help@subversion.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@subversion.apache.org Received: (qmail 86054 invoked by uid 99); 2 Feb 2018 05:40:36 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Feb 2018 05:40:36 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 6D40C1A6AD7; Fri, 2 Feb 2018 05:40:36 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.88 X-Spam-Level: * X-Spam-Status: No, score=1.88 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_SHORT=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id VnWzTJGwaTye; Fri, 2 Feb 2018 05:40:34 +0000 (UTC) Received: from mail-yw0-f182.google.com (mail-yw0-f182.google.com [209.85.161.182]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id A5A4A5F2C3; Fri, 2 Feb 2018 05:40:34 +0000 (UTC) Received: by mail-yw0-f182.google.com with SMTP id x62so12344140ywg.11; Thu, 01 Feb 2018 21:40:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=rAIwCwfLW7B4ptuXVD1CfKLCXAzezl68KTP+TivbyRg=; b=mvKv0bEsIjALOghM42tvdpZFNVuXqdqjvdX5mkiMO0XW4oMxQHGfJbN5epLkLVqdgb 64Lpc8AbaKSEeJ2Tz30brFmBqPUsuPYH+3ikf+5ENs4bC62ATWYC/g1yt8d1zMOlHaSw VdaUTzdE/GEYSPqdR+bibs2IIM7Rpm9KEDFE02IOBGsuh6DWPs+wP7lyl07zrvbVMH2A 7PuaafR4i4cfWkfZtoRovTSAYk0zJ6Tu9mYlaTIPZnwgAHE0esMUbv98s4A4Tbxywdei kZCVOpP+9CY98UxBKVYQrVXptN1oeP/l9KQZ6Do0U47b+cr8K7pXxk3USwh9J7HuVAhq /pGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=rAIwCwfLW7B4ptuXVD1CfKLCXAzezl68KTP+TivbyRg=; b=bl+9mcIEt9YsEf+6o7af57tiqR6xxngq9KudFejZ58LJ1zmqeOfiVg3XcH79zm2Xjg xrFIyxK2/AUr8ZGoiYvxUrhV11UN7tpQJv/VbL2tKDzt92/+O1BZRExOyBRxN/e4ojAa VozwgK5LXwudwx7VHKy36wpm/BQOTXFhwEAlzg86s1LLd/gKE8KrXa6iaDvaoCOuZxrf /w7MaorjJuzCVQcrwF1FuZW/ALTNpGiDg2XDrNlkFfq+foZcP2A1VaLS2EQs7KYPfqtJ dUyg+2P1u/BAVg6bN8CeLwtjZAz3SHdfozyl5n0+ZJk8pXxJ9ca7osLZtMLEFi7B7exr WL9A== X-Gm-Message-State: AKwxyte3KoAnhdhtPxMkK7Z/GnrLdHvsfeGNdYdoS/vCioJCJZwJxx64 MXzpCvaYrxx3xcrVNcnC/Ick01IA1q0vbtN/YSqH1Q== X-Google-Smtp-Source: AH8x224EoOLjUv3kWBA6hgefuqjjgY9SyXJxKEQamfbUesyIuWgjR1FXaUhSwKxaczlArvq1isaTaJ7/dNQg6s9eA4E= X-Received: by 10.37.132.78 with SMTP id r14mr1792978ybm.366.1517550028006; Thu, 01 Feb 2018 21:40:28 -0800 (PST) MIME-Version: 1.0 References: <0cb91b9d-8672-2975-4b5f-a921e1727e03@sewingwitch.com> <3CDAB57F7ADD5234EEA3043C@172.27.234.209> In-Reply-To: <3CDAB57F7ADD5234EEA3043C@172.27.234.209> From: Troy Curtis Jr Date: Fri, 02 Feb 2018 05:40:16 +0000 Message-ID: Subject: Re: mailer.py commit says TypeError: must be unicode, not str To: Kenneth Porter Cc: Subversion Development , users@subversion.apache.org Content-Type: multipart/alternative; boundary="089e0826ea90592a5b05643426d5" --089e0826ea90592a5b05643426d5 Content-Type: text/plain; charset="UTF-8" On Thu, Feb 1, 2018, 12:34 PM Kenneth Porter wrote: > [moving discussion to dev list as I think this is now the correct fix.] > > --On Wednesday, January 31, 2018 7:40 PM -0800 Kenneth Porter > wrote: > > > --On Wednesday, January 31, 2018 7:23 PM -0800 Kenneth Porter > > wrote: > > > >> fp = builtins.open(file, 'w+') # avoid namespace clash with > >> # trimmed-down svn_fs_open() > > > > I'm now thinking the problem is in the open call, and that I'm somehow > > getting a Python 3 open function even though I've got Python 2.7 > > installed. Should the mode be 'wb' instead of 'w+'? That would insure > > that the raw data from the Subversion object is getting dumped into the > > temporary fle without interpretation. I don't understand why update > > (denoted by the plus) is wanted. The temp file isn't being read from. > > > That seems strange, for py3 sure, but certainly odd on py2. Perhaps your locale is set to utf8? I'll have to research to see if that even makes sense. Proposed edit to fs.py: Change 'w+' to 'wb' when copying svn stream object > to temporary file. Update isn't needed, and the code just needs to dump the > raw data into a file for the external diff to access, so no > encoding/decoding should occur. Hence we should open the file in binary > mode. I just tested this edit and it seems to cure the problem. > > It looks like this line is the same since it was originally added in > r843330 and hasn't changed in Troy's swig-py3 branch. > I've been leaning heavily on the test coverage for validating my py3 updates. At first glance it looks like this FileDiff isn't referenced in any existing test. I'll add a test and confirm the behavior, and then test with your fix, unless you'd like to do so. Troy > >From my initial report in the users list: > > > > > I'm using mailer.py in my post-commit hook and it's throwing a Unicode type > error during the diff phase. Digging through the source code, I figured out > that it's happening during the creation of the two temporary files for > diff'ing. Somehow the output file is getting opened in Unicode text mode > but the input source (the Subversion object stream) is a raw byte stream. > The write call fails. > > OS: CentOS 7.4 > subversion-python-1.7.14-11.el7_4.x86_64 > python-2.7.5-58.el7.x86_64 > > > --089e0826ea90592a5b05643426d5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On Thu, Feb 1, 2018, 12= :34 PM Kenneth Porter <shiva@se= wingwitch.com> wrote:
[movin= g discussion to dev list as I think this is now the correct fix.]

--On Wednesday, January 31, 2018 7:40 PM -0800 Kenneth Porter
<shiva@sewing= witch.com> wrote:

> --On Wednesday, January 31, 2018 7:23 PM -0800 Kenneth Porter
> <shiva@s= ewingwitch.com> wrote:
>
>>=C2=A0 =C2=A0 =C2=A0fp =3D builtins.open(file, 'w+') # avoi= d namespace clash with
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # trimmed-down = svn_fs_open()
>
> I'm now thinking the problem is in the open call, and that I'm= somehow
> getting a Python 3 open function even though I've got Python 2.7 > installed. Should the mode be 'wb' instead of 'w+'? Th= at would insure
> that the raw data from the Subversion object is getting dumped into th= e
> temporary fle without interpretation. I don't understand why updat= e
> (denoted by the plus) is wanted. The temp file isn't being read fr= om.



That seems strange, for py3 sure, but certainly odd = on py2. Perhaps your locale is set to utf8? I'll have to research to se= e if that even makes sense.=C2=A0

Proposed edit to fs.py: Change 'w+' to 'wb' when copying sv= n stream object
to temporary file. Update isn't needed, and the code just needs to dump= the
raw data into a file for the external diff to access, so no
encoding/decoding should occur. Hence we should open the file in binary
mode. I just tested this edit and it seems to cure the problem.

It looks like this line is the same since it was originally added in
r843330 and hasn't changed in Troy's swig-py3 branch.

I've been leaning heavily on the test cove= rage for validating my py3 updates. At first glance it looks like this File= Diff isn't referenced in any existing test. I'll add a test and con= firm the behavior, and then test with your fix, unless you'd like to do= so.=C2=A0

Troy


>From my initial report in the users list:

<https://svn.haxx.se/users/archive-2018-01/00= 94.shtml>
<https://svn.haxx.se/users/archive-2018-02/00= 00.shtml>

I'm using mailer.py in my post-commit hook and it's throwing a Unic= ode type
error during the diff phase. Digging through the source code, I figured out=
that it's happening during the creation of the two temporary files for<= br> diff'ing. Somehow the output file is getting opened in Unicode text mod= e
but the input source (the Subversion object stream) is a raw byte stream. The write call fails.

OS: CentOS 7.4
subversion-python-1.7.14-11.el7_4.x86_64
python-2.7.5-58.el7.x86_64


--089e0826ea90592a5b05643426d5--