From dev-return-39649-archive-asf-public=cust-asf.ponee.io@subversion.apache.org Mon Oct 21 02:11:53 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 3C80D180629 for ; Mon, 21 Oct 2019 04:11:53 +0200 (CEST) Received: (qmail 85348 invoked by uid 500); 21 Oct 2019 02:11:52 -0000 Mailing-List: contact dev-help@subversion.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@subversion.apache.org Received: (qmail 85338 invoked by uid 99); 21 Oct 2019 02:11:51 -0000 Received: from Unknown (HELO mailrelay1-lw-us.apache.org) (10.10.3.42) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Oct 2019 02:11:51 +0000 Received: from [172.23.17.71] (89-233-126-4.dynamic.t-2.net [89.233.126.4]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id 3B3B25A6D for ; Mon, 21 Oct 2019 02:11:51 +0000 (UTC) Subject: Re: SWIG Python bindings build with SWIG 4.0 To: dev@subversion.apache.org References: <1f0a5007-0b0d-40d8-914d-33c6ad47c4d5@www.fastmail.com> <294a29d1-fa34-3616-983e-4a377b19fc7a@apache.org> <1aad152f-31ae-4c3e-936e-8da910c11ed9@www.fastmail.com> <0489b8ff-48a9-97db-269f-026c21218b31@apache.org> <20191017210136.32clbc5bbmqx3fdu@tarpaulin.shahaf.local2> <8172d5c4-01f7-d096-de1f-c768da6889eb@apache.org> <97e405ba-aa2c-887b-bd07-8887b493faf1@apache.org> <44f431a9-06d8-7369-ef61-29150e608c3e@apache.org> <308870d8-7127-c9f8-365b-7b0b9f52a4f0@apache.org> <1d22b36e-767d-f1c8-4084-c9225418a5e1@poem.co.jp> From: =?UTF-8?Q?Branko_=c4=8cibej?= Autocrypt: addr=brane@apache.org; prefer-encrypt=mutual; keydata= mQINBFG3qpMBEACi+jRQDd2TiYeAxVgrLZ3cyyuGOFSMh4nCyUOG9BwXC69cDLH48RcE0Mpu TFTGlfdokz6JgLKU3uqShPXiflrL6JIVnJX4rTEKRzFNkcS6Zq0PfNRnFnkwiD2KIzyAG8XE y0c1Bt7hqZ5dfXaC1b7Xo+1cnlqjdLAOnr1ruTrtfQ5sO81p9jYtARVa+iVmf8bs/FvC9Yn2 QtEDtuUfUUHx2bnB9vmh8tOjErfIcWtzCPt8uTUkmiszlkRMiB5/X97oqXlX/5dSQWE9m4M5 6Fc9ixIrmCwkF515RLrCNTv/YAtmpu4VaB0rxgTuSku0cVk83xSMrH2hNFx1fAeYBZpwp2GL ONlTy3D2N+BjWXjEUE9baGOoYM7QUbAdj4JMstSByppaAi4AiG9+raxknTWtWt2IT9LHW7Pu i6S3k4WL5jmTdQKqNQ9/+vRqiSVsA98yHQLa+s19IYh4F7WIfo2lzBAn06HEntpKS9TtV20o JyMBLOVqQP1dARWRfB0xIxGtbI61CfjEhCeG8H+UynCrHkUxgUoKsXXkI/JxsIMZ3TivFj3U MJVur7KVwg/isqqaEyMfUnCrXJxexZp8kuTjkzzvDKfYs0vHJezPQYhlqBLkK2w9VzktGjA7 lb+TO69bEyPOcBjVsCtrdYVc442/Z37G+1UV5+1X06m14Pt9UQARAQABtCBCcmFua28gxIxp YmVqIDxicmFuZUBhcGFjaGUub3JnPokCOgQTAQoAJAIbAwULCQgHAwUVCgkICwUWAgMBAAIe AQIXgAUCUbetMwIZAQAKCRAbymWGo0eUP2tOD/9KOLYfxwTcGV/Nj3lnKE4Y4gRl0r4cfnWm 1/2KyPYVsmQ8vWRUZxjuVHAvZrAkTBvlu+CVzrCWEEpCzQC/jki0xkPQchTEU2XOHQ6PzkXB 17o1NSSu/vyKynh0pXMRTHm4wZodzUw/tHn/Ism5QyRyhlYUP4mVX8v2hbN+stkJHrkdVBPm FspnFidhulUP5hr+LWz2qd+Ab8MOn3+x25jsGE8yaUiqmNdrmq/trvHPGThySa4Hz0uEkhfP K2knc6PpV5GTbeRn/J1eu17xVgXYVgko35Qwz5s/LRat+5R79tgBAL9SKFybCVBPr6/1Zp4u w9b0NcHW6t3aQHCxv8iEqxrJ7UIDhh/hXh4no0vzpPR1Cgjn6fK997WrpUyaAtlnbSH5QGad YY9rpFka3o2Gj+f+cr75hq6c7DnNJo94eGw9L0JEjfgordi8UkWErGOklnGf8N8brlVG0TdW 7KOz60m1E3UzIwd2lQd9a0zd8Mqrmn2MMPdJt4EpKQWaJsoK+FOdEBX0Ezm3StEXufe1IOG9 DihVcOnsx/G6aTS9GyKjURVt0jDB4wsVSzsRHYHmQpw7/ekvHFNKZS5yMNwSt2X/Szmk4GmV 69gaI79kf8VD87xwE31p0s0uVIVp7MTOTEYT5HUh5Rz6Rr66+vg9qgN1enMj5sh4f8krXgRR wLkCDQRRt6qTARAAnxIdGqDTC2FU9AE2ElT/m/Hs/57BwqUUb8qod3mJ6Qkp7PpHCBnvtbwm krrCsJl5rR1fliton6qoJUNCSfmcfeujcU8Be+q75rNZxIWi6AjMmyrjyMp9JIO7g/7+VYmL dm9c1wRn4QDnIKxl7qMPz9q8/OF6BGEMEW4zRL8rHvM7CCapOikHUKKq7GnZMVyYbue6KUTA Tczxjt6E9Av1QDnnW9zbW56jqUKdgpNek/bSTuef2xYEDzIzFPQREyw8E/C3xx8zZfOJ0+XV s1n39GLp3vugP5IBNE2pgqcyFtKISj1pVJgDr7zXjD92ZGS8xgqDxePTuf1LcCwd65BJNVVK IFsFicvBVhdslCZ7l8jkCuZAzYoFJZthUKuuJg1n7HYi8XLifZmun9Z3fbM5gk9/vA1rXsWt An597BACKDUkWA5tOb3Si4/MaRDiZYvzplHGc4sTn4aBIj3VFGGFNlOUPFLWjZLHdudNOBGj 3eIlz/DQZh/mwNGn5g98c3xehHnWxcXa0PsN2Xl1iRM2dec8drEVVRYaWPcOmGhKfqnlwl2z OeuST2TMcWhxKshVimR9eSt5pX1oGOD9PZ9V0gQDIr4d35UjQaW5ABCWbgTd7e3yPTlHoWx4 qyv+YoxEf6AlQ4nvE+q1s4wRBs/eNVQsROnYmhKhYPZUsDE6EocAEQEAAYkCHwQYAQoACQUC UbeqkwIbDAAKCRAbymWGo0eUPwd6D/92i5LBHSluiBdnzYH3kYlkIMjhy3lcqtxb/TWV1X/z CVpaZkEXvL9NQ44ZqfiOFB8fnaJvy+9rfIL3MwHKLVHOjsurBRP2DJ8H/EI6QuZV//Nxh66A dicXlE5SSiKQ5KcIH+eqZHa4XjVeXGeNZummrlhOv3ItKXETVhh2qeIQ/7zCjuw5rQk606+2 isg6cs4Nwtie1rXQ1KFtkTNQqWfqyM4PrEP9Bq5pWBQVkcxDsxk1Yj3A8L80IY3Hzwm8nRlq F+HkD/0IPgHICVDyiOB4XZtqVk+DHNOolCcdrFSXOcwt+qwD5zk4p0hdHKHagAPGBDXS8shm k2vaUDbKMUoVDdj579Jtp4tNOoVEEqqXspT995w7+ckbHGoQhFlSxCwtaXCr/8wwdwcCA2eO w0aLYrU04EbnH7Ryj4aTjsBGvJdmyZQT8/lTj5VARbEkNXTdTOs61pebDliyWtcF9Uz9b44p cLNniphcBO4SP/IMlEh8pBAJ1C2QlD4G90iJ1WK0MsJsUDix9Vb5s1AE6WA/Ss1iPCOdhhif eToCAwoobIipoxUZF2ik3oESskmMDolpVBiaPaFg+YPtNp/53dLap7jBNRNgyKXaGJAZaolp L+9hCU1EOWswqusDHDFSRUuYOXfuXZJxcbQUTnhQhRbvSDy3tDMRGd252Ur1sCOU5g== Organization: The Apache Software Foundation Message-ID: <75d512fe-3918-34aa-f7d9-c5fc676fd4f2@apache.org> Date: Mon, 21 Oct 2019 04:11:49 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Thunderbird/68.1.2 MIME-Version: 1.0 In-Reply-To: <1d22b36e-767d-f1c8-4084-c9225418a5e1@poem.co.jp> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-GB On 21.10.2019 01:15, Yasuhito FUTATSUKI wrote: > On 2019/10/20 21:58, Branko Čibej wrote: >> On 20.10.2019 06:35, Yasuhito FUTATSUKI wrote: >>> On 2019/10/20 8:37, Branko Čibej wrote: >>>> On 20.10.2019 01:10, Yasuhito FUTATSUKI wrote: >>>>> On 2019/10/20 6:59, Branko Čibej wrote: >>>>>> On 19.10.2019 23:06, Branko Čibej wrote: >>>>>>> On 19.10.2019 19:55, Branko Čibej wrote: >>>>>>>> On 19.10.2019 11:45, Yasuhito FUTATSUKI wrote: >>>>>>>>> On 2019/10/18 8:39, Branko Čibej wrote: >>>>>>>>>> On 17.10.2019 23:46, Branko Čibej wrote: >>>>>>>>>>> On 17.10.2019 23:01, Daniel Shahaf wrote: >>>>>>>>>>>> Branko Čibej wrote on Thu, Oct 17, 2019 at 22:56:29 +0200: >>>>>>>>>>>>> On 17.10.2019 22:37, Daniel Shahaf wrote: >>>>>>>>>>>>>> Branko Čibej wrote on Thu, 17 Oct 2019 06:49 +00:00: >>>>>>>>>>>>>>> On 16.10.2019 10:01, Daniel Shahaf wrote: >>>>>>>>>>>>>>>> Which reminds me: it'd be nice to set up a buildbot to >>>>>>>>>>>>>>>> run the >>>>>>>>>>>>>>>> build and >>>>>>>>>>>>>>>> test process under Python 3. Any committer can edit the >>>>>>>>>>>>>>>> buildbot >>>>>>>>>>>>>>>> scripts[1], but the question is which of the buildbot >>>>>>>>>>>>>>>> slaves >>>>>>>>>>>>>>>> has Python 3 >>>>>>>>>>>>>>>> installed? >>>>>>>>>>>>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Been there, done that, bought the DVD. >>>>>>>>>>>>>> Why are svnadmin_tests 69 and tree_conflict_tests 26 passing >>>>>>>>>>>>>> on that >>>>>>>>>>>>>> bot?  They (currently) fail under python3. >>>>>>>>>>>>> I don't know. It's possible, I suppose, that the >>>>>>>>>>>>> activation of >>>>>>>>>>>>> the >>>>>>>>>>>>> python3 virtual environment has no effect on the test >>>>>>>>>>>>> driver. It >>>>>>>>>>>>> might >>>>>>>>>>>>> not be a bad idea to have the tests print the Python version >>>>>>>>>>>>> in the >>>>>>>>>>>>> summary and possibly in the log of every test case. >>>>>>>>>>>> What's the value of $(PYTHON) in Makefile?  That's what the >>>>>>>>>>>> «check» >>>>>>>>>>>> target uses. >>>>>>>>>>> Yep, apparently that's the bug ... I'm testing script changes >>>>>>>>>>> now, >>>>>>>>>>> along >>>>>>>>>>> with r1868566 for good measure. >>>>>>>>>> I've set up this: >>>>>>>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3-swig >>>>>>>>>> >>>>>>>>>> It will build and test the core libraries and swig-py bindings >>>>>>>>>> with >>>>>>>>>> Python 3 on the swig-py3 branch. >>>>>>>>>> >>>>>>>>>> I hope ... build #0 running as we speak. >>>>>>>>> Unfortunately build #2, which ran after upgrading SWIG and >>>>>>>>> Python, >>>>>>>>> failed >>>>>>>>> to build SWIG Python bindings because of SWIG 4.0, as reported on >>>>>>>>> SVN-4818. >>>>>>>>> >>>>>>>>> This also affects on trunk with SWIG 4.0. >>>>>>>>> (e.g. >>>>>>>>> https://ci.apache.org/builders/svn-x64-macosx-full/builds/2418) >>>>>>>>> >>>>>>>>> With attached patch on trunk >>>>>>>>> (trunk_build_with_swig4_patch.txt) and >>>>>>>>> on swig-py3 (swig_py3_build_with_swig4_patch.txt), *.so can be >>>>>>>>> produced, >>>>>>>>> but the modules don't work correctly. >>>>>>>>> >>>>>>>>> It seems they were caused by incompatibility of Python code for >>>>>>>>> proxy >>>>>>>>> object generated by SWIG, and it can not be resolved so >>>>>>>>> simple.... >>>>>>>>> (importlib module vs simply use 'import', absense of >>>>>>>>> _swig_setattr() >>>>>>>>> and _swig_getattr(), etc.) >>>>>>>> My bad. I downgraded to Swig 3.0.12 and restarted the failed >>>>>>>> builds. >>>>>>> >>>>>>> Hm, that did not really help. On the swig-py3 branch: >>>>>>> >>>>>>> /bin/sh: line 1: 62481 Segmentation fault: 11 >>>>>>> /srv/virtualenv-3/bin/python >>>>>>> /srv/buildbot/slave/svn-x64-macosx-local-python3-swig/build/subversion/bindings/swig/python/tests/run_all.py >>>>>>> >>>>>>> >>>>>>> make: *** [check-swig-py] Error 139 >>>>>>> >>>>>>> >>>>>>> So, the Python bindings seem to have crashed the Python 3 >>>>>>> interpreter on >>>>>>> macOS? >>>>>> >>>>>> Yes, they did: >>>>>> >>>>>> Process 28278 stopped >>>>>> * thread #2, queue = 'com.apple.main-thread', stop reason = >>>>>> EXC_BAD_ACCESS (code=1, address=0x48) >>>>>>        frame #0: 0x00007fff373ab746 Python`PyErr_Restore + 61 >>>>>> Python`PyErr_Restore: >>>>>> ->  0x7fff373ab746 <+61>: movq   0x48(%rbx), %rdi >>>>>>        0x7fff373ab74a <+65>: movq   0x50(%rbx), %r13 >>>>>>        0x7fff373ab74e <+69>: movq   0x58(%rbx), %r12 >>>>>>        0x7fff373ab752 <+73>: movq   %r15, 0x48(%rbx) >>>>>> Target 0: (Python) stopped. >>>>>> >>>>>> frame #3: 0x0000000103800fea _core.so`PyInit__core at >>>>>> core.c:40158:12 >>>>>>       40155      m = Py_InitModule((char *) SWIG_name, SwigMethods); >>>>>>       40156    #endif >>>>>>       40157 >>>>>> -> 40158      md = d = PyModule_GetDict(m); >>>>>>       40159      (void)md; >>>>>>       40160 >>>>>>       40161      SWIG_InitializeModule(0); >>>>>> (lldb) print m >>>>>> (PyObject *) $0 = 0x000000010279ad70 >>>>>> (lldb) print *m >>>>>> (PyObject) $1 = { >>>>>>      ob_refcnt = 785 >>>>>>      ob_type = 0x000000010026ea30 >>>>>> } >>>>> >>>>> I guess the _core module linked incompatible Python library to >>>>> execute. >>>>> >>>>> on >>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3-swig/builds/6/steps/Build/logs/stdio >>>>> >>>>> >>>>> : >>>>> [[[ >>>>> checking for compiling Python extensions... clang >>>>> checking for linking Python extensions... clang -bundle -undefined >>>>> dynamic_lookup -isysroot >>>>> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk >>>>> >>>>> >>>>> -F/usr/local/opt/python/Frameworks -framework Python >>>>> checking for linking Python libraries... -bundle -undefined >>>>> dynamic_lookup -F/usr/local/opt/python/Frameworks -framework Python >>>>> ]]] >>>> >>>> >>>> Good catch. This certainly looks correct, it's where Homebrew installs >>>> python. However ... >>>> >>>> $ otool -L /opt/mine/svn-swig-py3/lib/svn-python/libsvn/_core.so >>>> /opt/mine/svn-swig-py3/lib/svn-python/libsvn/_core.so: >>>>      /System/Library/Frameworks/Python.framework/Versions/2.7/Python >>>> (compatibility version 2.7.0, current version 2.7.10) >>>>           ... >>>> >>>> >>>> That is wrong. Looks like we'll have to set DYLD_LIBRARY_PATH so >>>> that it >>>> finds the Python 3 installation. >> >> (Actually that would be DYLD_FRAMEWORK_PATH.) >> >>> ... or have to use Run-Path Dependent Libraries feature on link time, >>> if DYLD_LIBRARY_PATH cannot work well due to SIP. >>> >>> https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/RunpathDependentLibraries.html >>> >>> >>> >>> The link option in log above comes from result of >>> 'build/get-py-info.py --link', so should we tweak build/get-py-info.py, >>> or add option to configure to pass Python dynamic library path? >> >> >> Using RPATH for the Python framework on macOS would be the right >> solution, but it doesn't solve the problem. It looks like we need some >> magic incantation on macOS to explicitly pick the Python 3 framework; >> using just the -F option doesn't work as expected, the path to the >> system default Python 2.7 framework is hard-coded in to the extension >> libraries. > > Umm.... it seems extension libraries actually don't need to link > "Python" dynamic library. > > e.g. > $ otool -L `python -c 'import _socket;print(_socket.__file__)'` > /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_socket.so: > >         /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, > current version 1252.250.1) You're right. Fixed in r1868674, this makes the Python bindings tests work on my laptop, with both the system Python 2 and the Homebrewed Python 3. I've forced the tests to run on the buildslave. -- Brane