mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Naveen Swamy <mnnav...@gmail.com>
Subject Call to Help Improve Build times.
Date Wed, 23 Aug 2017 03:48:22 GMT
I have noticed that our build jobs have been taking about 2+ hr to build
and test for every PR and merge. I have observed that the windows build and
unit tests take about 30 minutes to build and 50 minutes to run the unit
tests. There is also CUDA7.5 + cuDNN and MKL builds that take 30+ minutes
each respectively to just build. Though they run in parallel there is not
always enough capacity to handle these.

Can someone who has expertise in these areas to help explain what part of
these builds are taking so long and help reduce the build times.

One area I see is running the Python Windows GPU Unit tests that can be
parallelized for Python2 and Python3 on the same machine by using Conda
Environments(i think the slaves already have Conda Environments),

We are also considering to modify our PR builds to not build immediately
after a PR is submitted and instead run only sanity tests and wait for a
reviewer to tag it as "ready to build". This could alleviate some load off
and only run when the PR is really ready.

You can view the various builds time by clicking through the stages at the
top of this sample build.
https://builds.apache.org/blue/organizations/jenkins/incubator-mxnet/detail/v0.11.0/5/pipeline/49

yajiedesign@, can you please take a look to see if we can improve Windows
build/test times?


Thanks, Naveen

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message