mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tsuyoshi Ozawa <oz...@apache.org>
Subject Re: CI system seems to be using python3 for python2 builds
Date Thu, 28 Sep 2017 06:40:02 GMT
Thanks for your sharing information.

> 1. Looks like this backtrace has been present since long time, since this
> was not a test failure or build failure we never got notified about it. Here
> <https://builds.apache.org/view/Incubator%20Projects/job/incubator-mxnet/job/master/448/consoleFull>
> is the recent build log where back trace is present but build succeeded.

Oh, my... but it's very essential information. I need more survey.
To understand what is real cause of the problem(on program-side or
infra-side), I would like to run Jenkinsfile locally.
Please let me give a time for a while to check - including whether
#7658 is real cause(it looks not).

> 2. I don't think the default version of python on Ubuntu is 3.0, I logged
> into one of the apache slave and the default version of Python is 2.7.6

Ouch, the "python" command is an alias of python2.7 on Ubuntu as you
mentioned. Thanks for your pointing out.

I will share information here when I have a progress, so I appreciate
if you do so too.

Thanks,
- Tsuyoshi

On Thu, Sep 28, 2017 at 2:46 PM, Gautam <gautamnitc@gmail.com> wrote:
> Hi Ozawa,
>
>   Thanks for follow up.
>   Unfortunately I didn't get time to work on this today.
>
> However I have couple of points to mentions.
> 1. Looks like this backtrace has been present since long time, since this
> was not a test failure or build failure we never got notified about it. Here
> <https://builds.apache.org/view/Incubator%20Projects/job/incubator-mxnet/job/master/448/consoleFull>
> is the recent build log where back trace is present but build succeeded.
>
> 2. I don't think the default version of python on Ubuntu is 3.0, I logged
> into one of the apache slave and the default version of Python is 2.7.6
>
> 3. There has been slight change
> <https://github.com/apache/incubator-mxnet/pull/7658> in Jenkins file where
> we tried to parallelize python2 and 3 test run. I am not sure if it
> affects. I can probably scrub the build log and figure out if thats the
> case.
>
>
> Feel free to send the PR, if you have it ready.
>
>
> -Gautam
>
>
> On Wed, Sep 27, 2017 at 9:39 PM, Tsuyoshi Ozawa <ozawa@apache.org> wrote:
>
>> Hi Kumar,
>>
>> Thanks for looking into the issue. How is the progress of this problem?
>> Shouldn't we call /usr/bin/env python2 or python2.7 in following
>> source code instead of python since MXNet only supports python2
>> currently?
>> I think default version of python in Ubuntu is now python3, so it can
>> cause the problem.
>> If you have not yet done the work, I can create a PR for that in this
>> weekend.
>>
>> ./python/mxnet/__init__.py:#!/usr/bin/env python
>> ./python/mxnet/log.py:#!/usr/bin/env python
>> ./tests/nightly/dist_lenet.py:#!/usr/bin/env python
>> ./tests/nightly/dist_sync_kvstore.py:#!/usr/bin/env python
>> ./tests/nightly/multi_lenet.py:#!/usr/bin/env python
>> ./tests/nightly/test_kvstore.py:#!/usr/bin/env python
>> ./tools/coreml/mxnet_coreml_converter.py:#!/usr/bin/env python
>> ./tools/ipynb2md.py:#!/usr/bin/env python
>> ./tools/kill-mxnet.py:#!/usr/bin/env python
>> ./tools/launch.py:#!/usr/bin/env python
>> ./tools/parse_log.py:#!/usr/bin/env python
>>
>> On Wed, Sep 27, 2017 at 5:39 PM, Sunderland, Kellen <kellens@amazon.de>
>> wrote:
>> > Many thanks Gautam.
>> >
>> > On 9/26/17, 8:37 PM, "Kumar, Gautam" <gauta@amazon.com> wrote:
>> >
>> >     Hi Kellen,
>> >
>> >        This issue has been happening since last 3-4 days along with few
>> other test failure.
>> >     I am looking into it.
>> >
>> >     -Gautam
>> >
>> >     On 9/26/17, 7:45 AM, "Sunderland, Kellen" <kellens@amazon.de> wrote:
>> >
>> >         I’ve been noticing in a few failed builds that the stack trace
>> indicates we’re actually running python 3.4 in the python 2 tests. I know
>> the CI folks are working hard getting everything setup, is this a known
>> issue for the CI team?
>> >
>> >         For example: https://builds.apache.org/
>> blue/organizations/jenkins/incubator-mxnet/detail/PR-8026/3/pipeline/281
>> >
>> >         Steps Python2: MKLML-CPU
>> >
>> >         StackTrace:
>> >         Stack trace returned 10 entries:
>> >         [bt] (0) /workspace/python/mxnet/../../lib/libmxnet.so(_
>> ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7fadb8999aac]
>> >         [bt] (1) /workspace/python/mxnet/../../lib/libmxnet.so(_
>> ZN5mxnet7kvstore12KVStoreLocal12GroupKVPairsISt4pairIPNS_
>> 7NDArrayES4_EZNS1_19GroupKVPairsPullRspERKSt6vectorIiSaIiEERKS7_IS6_SaIS6_
>> EEPS9_PS7_ISD_SaISD_EEEUliRKS6_E_EEvSB_RKS7_IT_SaISN_EESG_PS7_ISP_SaISP_EERKT0_+0x56b)
>> [0x7fadba32c01b]
>> >         [bt] (2) /workspace/python/mxnet/../../lib/libmxnet.so(_
>> ZN5mxnet7kvstore12KVStoreLocal17PullRowSparseImplERKSt6vecto
>> rIiSaIiEERKS2_ISt4pairIPNS_7NDArrayES8_ESaISA_EEi+0xa6) [0x7fadba32c856]
>> >         [bt] (3) /workspace/python/mxnet/../../lib/libmxnet.so(MXKVStorePullRowSparse+0x245)
>> [0x7fadba18f165]
>> >         [bt] (4) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c)
>> [0x7fadde26cadc]
>> >         [bt] (5) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x1fc)
>> [0x7fadde26c40c]
>> >         [bt] (6) /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-
>> x86_64-linux-gnu.so(_ctypes_callproc+0x21d) [0x7fadde47e12d]
>> >         [bt] (7) /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-
>> x86_64-linux-gnu.so(+0xf6a3) [0x7fadde47e6a3]
>> >         [bt] (8) /usr/bin/python3(PyEval_EvalFrameEx+0x41d7) [0x48a487]
>> >         [bt] (9) /usr/bin/python3() [0x48f2df]
>> >
>> >         -Kellen
>> >         Amazon Development Center Germany GmbH
>> >         Berlin - Dresden - Aachen
>> >         main office: Krausenstr. 38, 10117 Berlin
>> >         Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
>> >         Ust-ID: DE289237879
>> >         Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
>> >
>> >
>> >
>> >
>> > Amazon Development Center Germany GmbH
>> > Berlin - Dresden - Aachen
>> > main office: Krausenstr. 38, 10117 Berlin
>> > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
>> > Ust-ID: DE289237879
>> > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
>>
>
>
>
> --
> Best Regards,
> Gautam Kumar

Mime
View raw message