mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tsuyoshi Ozawa <oz...@apache.org>
Subject Re: CI system seems to be using python3 for python2 builds
Date Thu, 28 Sep 2017 07:18:22 GMT
Hi Rahul,

Thanks for sharing the information and taking on the problem!
I will also try the option(using nosetests-2.7 in Jenkinsfile).

Regards,
- Tsuyoshi

On Thu, Sep 28, 2017 at 3:34 PM, Rahul Huilgol <rahulhuilgol@gmail.com> wrote:
> Hi Gautam,
>
> I see that ‘nosetests’ is the command used to run python2 tests. It looks
> like that’s being mapped to use python3. I’ve checked that this is the case
> on my Ubuntu instance. I need to use ‘nosetests-2.7’ to use python2 for the
> tests. Please check if this fix works in the build environment
> (slave/docker container) as well.
>
> The PR you refer to only parallelized tests that were running one after the
> other, this command was being used even before that PR.
>
> Regards,
> Rahul
>
> On Wed, 27 Sep 2017 at 22:46 Gautam <gautamnitc@gmail.com> wrote:
>
>> Hi Ozawa,
>>
>>   Thanks for follow up.
>>   Unfortunately I didn't get time to work on this today.
>>
>> However I have couple of points to mentions.
>> 1. Looks like this backtrace has been present since long time, since this
>> was not a test failure or build failure we never got notified about it.
>> Here
>> <
>> https://builds.apache.org/view/Incubator%20Projects/job/incubator-mxnet/job/master/448/consoleFull
>> >
>> is the recent build log where back trace is present but build succeeded.
>>
>> 2. I don't think the default version of python on Ubuntu is 3.0, I logged
>> into one of the apache slave and the default version of Python is 2.7.6
>>
>> 3. There has been slight change
>> <https://github.com/apache/incubator-mxnet/pull/7658> in Jenkins file
>> where
>> we tried to parallelize python2 and 3 test run. I am not sure if it
>> affects. I can probably scrub the build log and figure out if thats the
>> case.
>>
>>
>> Feel free to send the PR, if you have it ready.
>>
>>
>> -Gautam
>>
>>
>> On Wed, Sep 27, 2017 at 9:39 PM, Tsuyoshi Ozawa <ozawa@apache.org> wrote:
>>
>> > Hi Kumar,
>> >
>> > Thanks for looking into the issue. How is the progress of this problem?
>> > Shouldn't we call /usr/bin/env python2 or python2.7 in following
>> > source code instead of python since MXNet only supports python2
>> > currently?
>> > I think default version of python in Ubuntu is now python3, so it can
>> > cause the problem.
>> > If you have not yet done the work, I can create a PR for that in this
>> > weekend.
>> >
>> > ./python/mxnet/__init__.py:#!/usr/bin/env python
>> > ./python/mxnet/log.py:#!/usr/bin/env python
>> > ./tests/nightly/dist_lenet.py:#!/usr/bin/env python
>> > ./tests/nightly/dist_sync_kvstore.py:#!/usr/bin/env python
>> > ./tests/nightly/multi_lenet.py:#!/usr/bin/env python
>> > ./tests/nightly/test_kvstore.py:#!/usr/bin/env python
>> > ./tools/coreml/mxnet_coreml_converter.py:#!/usr/bin/env python
>> > ./tools/ipynb2md.py:#!/usr/bin/env python
>> > ./tools/kill-mxnet.py:#!/usr/bin/env python
>> > ./tools/launch.py:#!/usr/bin/env python
>> > ./tools/parse_log.py:#!/usr/bin/env python
>> >
>> > On Wed, Sep 27, 2017 at 5:39 PM, Sunderland, Kellen <kellens@amazon.de>
>> > wrote:
>> > > Many thanks Gautam.
>> > >
>> > > On 9/26/17, 8:37 PM, "Kumar, Gautam" <gauta@amazon.com> wrote:
>> > >
>> > >     Hi Kellen,
>> > >
>> > >        This issue has been happening since last 3-4 days along with few
>> > other test failure.
>> > >     I am looking into it.
>> > >
>> > >     -Gautam
>> > >
>> > >     On 9/26/17, 7:45 AM, "Sunderland, Kellen" <kellens@amazon.de>
>> wrote:
>> > >
>> > >         I’ve been noticing in a few failed builds that the stack trace
>> > indicates we’re actually running python 3.4 in the python 2 tests. I know
>> > the CI folks are working hard getting everything setup, is this a known
>> > issue for the CI team?
>> > >
>> > >         For example: https://builds.apache.org/
>> > blue/organizations/jenkins/incubator-mxnet/detail/PR-8026/3/pipeline/281
>> > >
>> > >         Steps Python2: MKLML-CPU
>> > >
>> > >         StackTrace:
>> > >         Stack trace returned 10 entries:
>> > >         [bt] (0) /workspace/python/mxnet/../../lib/libmxnet.so(_
>> > ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7fadb8999aac]
>> > >         [bt] (1) /workspace/python/mxnet/../../lib/libmxnet.so(_
>> > ZN5mxnet7kvstore12KVStoreLocal12GroupKVPairsISt4pairIPNS_
>> >
>> 7NDArrayES4_EZNS1_19GroupKVPairsPullRspERKSt6vectorIiSaIiEERKS7_IS6_SaIS6_
>> >
>> EEPS9_PS7_ISD_SaISD_EEEUliRKS6_E_EEvSB_RKS7_IT_SaISN_EESG_PS7_ISP_SaISP_EERKT0_+0x56b)
>> > [0x7fadba32c01b]
>> > >         [bt] (2) /workspace/python/mxnet/../../lib/libmxnet.so(_
>> > ZN5mxnet7kvstore12KVStoreLocal17PullRowSparseImplERKSt6vecto
>> > rIiSaIiEERKS2_ISt4pairIPNS_7NDArrayES8_ESaISA_EEi+0xa6) [0x7fadba32c856]
>> > >         [bt] (3)
>> /workspace/python/mxnet/../../lib/libmxnet.so(MXKVStorePullRowSparse+0x245)
>> > [0x7fadba18f165]
>> > >         [bt] (4)
>> /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c)
>> > [0x7fadde26cadc]
>> > >         [bt] (5) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x1fc)
>> > [0x7fadde26c40c]
>> > >         [bt] (6) /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-
>> > x86_64-linux-gnu.so(_ctypes_callproc+0x21d) [0x7fadde47e12d]
>> > >         [bt] (7) /usr/lib/p
>> <https://maps.google.com/?q=(7)+/usr/lib/p&entry=gmail&source=g>
>> ython3.4/lib-dynload/_ctypes.cpython-34m-
>> > x86_64-linux-gnu.so(+0xf6a3) [0x7fadde47e6a3]
>> > >         [bt] (8) /usr/bin/python3(PyEval_EvalFrameEx+0x41d7) [0x48a487]
>> > >         [bt] (9) /usr/bin/python3() [0x48f2df]
>> > >
>> > >         -Kellen
>> > >         Amazon Development Center Germany GmbH
>> > >         Berlin - Dresden - Aachen
>> > >         main office: Krausenstr. 38, 10117 Berlin
>> > >         Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
>> > >         Ust-ID: DE289237879
>> > >         Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
>> > >
>> > >
>> > >
>> > >
>> > > Amazon Development Center Germany GmbH
>> > > Berlin - Dresden - Aachen
>> > > main office: Krausenstr. 38, 10117 Berlin
>> > > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
>> > > Ust-ID: DE289237879
>> > > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
>> >
>>
>>
>>
>> --
>> Best Regards,
>> Gautam Kumar
>>

Mime
View raw message