mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sheng Zha <szha....@gmail.com>
Subject Re: [VOTE] Release Apache MXNet (incubating) version 1.5.0.rc0
Date Wed, 19 Jun 2019 00:26:00 GMT
This vote has been closed. We will make another tag and start vote again.

-sz

> On Jun 18, 2019, at 5:24 PM, Lin Yuan <apeforest@gmail.com> wrote:
> 
> With the PR https://github.com/apache/incubator-mxnet/pull/15213 I could
> verify that building Horovod is successful with MXNet built from source. So
> I will remove my pervious -1 vote.
> 
> Best,
> 
> Lin
> 
>> On Tue, Jun 18, 2019 at 2:10 PM Junru Shao <junrushao1994@gmail.com> wrote:
>> 
>> Dear community,
>> 
>> I am happy to share some results with regard to commit 83d2c2d0e (PR
>> #14192, link: https://github.com/apache/incubator-mxnet/pull/14192) that
>> Pedro mentioned that causes regression.
>> 
>> First, using the exact model that Pedro provides, we did rigorous profiling
>> and found out that the PR #14192 slows it down by 7.26 ms (from 235.65 ms
>> to 242.91 ms).
>> 
>> Then, we submitted a following up PR #15262 (link:
>> https://github.com/apache/incubator-mxnet/pull/15262) to fix the
>> regression. By applying the patch to commit 83d2c2d0e, we could verify that
>> we get comparable performance. Please refer to the PR if you are interested
>> in our experiment.
>> 
>> That is to say, regression caused by commit 83d2c2d0e should have been
>> addressed. Please let me know if there is any future issues.
>> 
>> Thank you so much,
>> Junru
>> 
>> On Thu, Jun 13, 2019 at 3:05 PM Pedro Larroy <pedro.larroy.lists@gmail.com
>>> 
>> wrote:
>> 
>>> I reach you in private, the model is not public. We should be able to
>>> see this problem in a public model using LSTM I think.
>>> 
>>> 
>>> On Thu, Jun 13, 2019 at 11:15 AM Junru Shao <junrushao1994@gmail.com>
>>> wrote:
>>>> 
>>>> Hi Pedro,
>>>> 
>>>> Thanks for brining this up!
>>>> 
>>>> Could you provide your model so that we can dig into this?
>>>> 
>>>> Thanks,
>>>> Junru
>>>> 
>>>> On Thu, Jun 13, 2019 at 10:33 Pedro Larroy <
>> pedro.larroy.lists@gmail.com
>>>> 
>>>> wrote:
>>>> 
>>>>> I have isolated some of the commits that are causing performance
>>>>> regressions in wavenet like models:
>>>>> 
>>>>> Title: 83d2c2d0e:[MXNET-1324] Add NaiveRunGraph to imperative utils
>>>>> (#14192)
>>>>> 
>>>>> Causes a regression making hybridize with static slower using GPU
>>>>> inference.
>>>>> 
>>>>> [0f63659be5070af218095a6a460427d2a1b67aba] add a compiler flag to use
>>>>> int64 as tensor size (#14570)
>>>>> 
>>>>> Causes overall regressions in CPU inference.
>>>>> 
>>>>> 
>>>>> Pedro.
>>>>> 
>>>>> On Wed, Jun 12, 2019 at 11:52 AM Lai Wei <royweilai@gmail.com>
>> wrote:
>>>>>> 
>>>>>> Hi @dev,
>>>>>> 
>>>>>> I am canceling the vote as the issue Lin discovered require a
>> fix[1]
>>> and
>>>>>> the solution is not ready yet.
>>>>>> It's a general problem when building from source with MXNet, not
>> only
>>>>>> impacting horovod use cases.  Any help is appreciated.
>>>>>> 
>>>>>> Other issues we are tracking:
>>>>>> 1. Regression on hybridize with static_alloc. (not a blocker for
>> now)
>>>>>> 2. Scala doc issue [2], already merged in master, need to backport
>> to
>>>>> 1.5.x
>>>>>> 
>>>>>> Thanks for everyone's help! Please let us know if there is any
>> other
>>>>> issue
>>>>>> with 1.5.0
>>>>>> 
>>>>>> [1] https://github.com/apache/incubator-mxnet/pull/15213
>>>>>> [2] https://github.com/apache/incubator-mxnet/pull/15216
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Best Regards
>>>>>> 
>>>>>> Lai
>>>>>> 
>>>>>> 
>>>>>> On Tue, Jun 11, 2019 at 5:04 PM Pedro Larroy <
>>>>> pedro.larroy.lists@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> Tested with CPU, 2.6x slower. comparing master vs 1.4.1.
>>>>>>> 
>>>>>>> Looks like a general regression.
>>>>>>> 
>>>>>>> 
>>>>>>> On Tue, Jun 11, 2019 at 2:31 PM Lai Wei <royweilai@gmail.com>
>>> wrote:
>>>>>>>> 
>>>>>>>> Hi guys,
>>>>>>>> 
>>>>>>>> Thanks for the updates. Currently, we are able to confirm
Lin's
>>> issue
>>>>>>> with
>>>>>>>> Horovod, and there is a fix pending. [1]
>>>>>>>> Will update later today to see if we need to cancel this
vote
>>> for the
>>>>>>> fix.
>>>>>>>> 
>>>>>>>> As for the hybridize with static alloc performance regression.
>>> IMO it
>>>>>>> does
>>>>>>>> not need to be a blocker if we have the following speed order.
>>>>>>>> 1.5.0 w/o static > 1.5.0 w/ static  > 1.4.1 w/ static
> 1.4.1
>> w/o
>>>>> static
>>>>>>>> and it will be great to know the following to better make
a
>>> decision
>>>>> on
>>>>>>>> whether this should block the release.
>>>>>>>> 1) if this is a model specific or a general regression.
>>>>>>>> 2) if this is platform specific or general (w/ or w/o CUDA,
w/
>>> or w/o
>>>>>>>> MKLDNN)
>>>>>>>> 
>>>>>>>> 
>>>>>>>> [1]https://github.com/apache/incubator-mxnet/pull/15213
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> 
>>>>>>>> Best Regards
>>>>>>>> 
>>>>>>>> Lai
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Tue, Jun 11, 2019 at 1:46 PM Zhi Zhang <
>> zhreshold@apache.org>
>>>>> wrote:
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On 2019/06/11 18:53:56, Pedro Larroy <
>>> pedro.larroy.lists@gmail.com
>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>>> The stack trace doesn't seem to come from MXNet,
do you
>> have
>>> more
>>>>>>> info?
>>>>>>>>>> 
>>>>>>>>>> On Tue, Jun 11, 2019 at 11:46 AM Zhi Zhang <
>>> zhreshold@apache.org
>>>>>> 
>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On 2019/06/11 17:36:09, Pedro Larroy <
>>>>> pedro.larroy.lists@gmail.com
>>>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>>>>> A bit more background into this:
>>>>>>>>>>>> 
>>>>>>>>>>>> While tuning a model using LSTM and convolutions
we
>> find
>>> that
>>>>>>> using
>>>>>>>>>>>> hybridize with static_alloc and static_shape
is 15%
>>> slower
>>>>> in the
>>>>>>>>>>>> latest revision vs in version 1.4.1 in which
using
>>> hybridize
>>>>> with
>>>>>>>>>>>> static_alloc and static_shape is 10% faster
than
>> without.
>>>>>>>>>>>> 
>>>>>>>>>>>> Overwall we are still 33% faster when comparing
master
>> to
>>>>> 1.5.
>>>>>>>>>>>> 
>>>>>>>>>>>> Let me know if you think this is a release
blocker or
>>> not.
>>>>>>>>>>>> 
>>>>>>>>>>>> Pedro.
>>>>>>>>>>>> 
>>>>>>>>>>>> On Mon, Jun 10, 2019 at 4:51 PM Pedro Larroy
>>>>>>>>>>>> <pedro.larroy.lists@gmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -1
>>>>>>>>>>>>> 
>>>>>>>>>>>>> We found a performance regression vs
1.4 related to
>>>>> CachedOp
>>>>>>> which
>>>>>>>>>>>>> affects Hybrid forward, which we are
looking into.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Pedro.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Mon, Jun 10, 2019 at 4:33 PM Lin Yuan
<
>>>>> apeforest@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -1 (Tentatively until resolved)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I tried to build MXNet 1.5.0 from
source and pip
>>> install
>>>>>>> horovod
>>>>>>>>> but got
>>>>>>>>>>>>>> the following error:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Reproduce:
>>>>>>>>>>>>>> 1) cp make/config.mk .
>>>>>>>>>>>>>> 2) turn on USE_CUDA, USE_CUDNN, USE_NCCL
>>>>>>>>>>>>>> 3) make -j
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> MXNet can build successfully.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 4) pip install horovod
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> 
>> /home/ubuntu/src/incubator-mxnet/python/mxnet/../../include/mkldnn/mkldnn.h:55:28:
>>>>>>>>>>>>>> fatal error: mkldnn_version.h: No
such file or
>>> directory
>>>>>>>>>>>>>>    compilation terminated.
>>>>>>>>>>>>>>    INFO: Unable to build MXNet plugin,
will skip
>> it.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I did not change any setting of MKLDNN
in my
>>> config.mk.
>>>>> I am
>>>>>>>>> building on
>>>>>>>>>>>>>> DLAMI base 18.0 which is Ubuntu 16.04
and CUDA 10.0
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Lin
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sat, Jun 8, 2019 at 5:39 PM shiwen
hu <
>>>>>>> yajiedesign@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> +1
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Lai Wei <royweilai@gmail.com>
于2019年6月9日周日
>>> 上午4:12写道:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Dear MXNet community,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> This is the 3-day vote to
release Apache MXNet
>>>>>>> (incubating)
>>>>>>>>> version
>>>>>>>>>>>>>>> 1.5.0.
>>>>>>>>>>>>>>>> Voting on dev@ will start
June 8,
>>> 23:59:59(PST)  and
>>>>>>> close
>>>>>>>>> on June 11,
>>>>>>>>>>>>>>>> 23:59:59.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 1) Link to release notes:
>>>>>>>>>>>>>>>> 
>>>>>>>>> 
>>>>> 
>> https://cwiki.apache.org/confluence/display/MXNET/1.5.0+Release+Notes
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 2) Link to release candidate:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>> 
>>> https://github.com/apache/incubator-mxnet/releases/tag/1.5.0.rc0
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 3) Link to source and signatures
on apache dist
>>>>> server:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>> 
>>> https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.5.0.rc0/
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Please remember to TEST first
before voting
>>>>> accordingly:
>>>>>>>>>>>>>>>> +1 = approve
>>>>>>>>>>>>>>>> +0 = no opinion
>>>>>>>>>>>>>>>> -1 = disapprove (provide
reason)
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Best Regards
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Lai
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> -1. Built from source, import mxnet in python
cause
>>> Segfault.
>>>>>>>>>>> 
>>>>>>>>>>> back trace:
>>>>>>>>>>> 
>>>>>>>>>>> Thread 1 "python3" received signal SIGSEGV, Segmentation
>>> fault.
>>>>>>>>>>> 0x00007fff3e8a9f20 in ?? ()
>>>>>>>>>>> (gdb) bt
>>>>>>>>>>> #0  0x00007fff3e8a9f20 in ?? ()
>>>>>>>>>>> #1  0x00007fffebbf440c in ReadConfigFile(Configuration&,
>>>>>>>>>>> std::__cxx11::basic_string<char, std::char_traits<char>,
>>>>>>>>>>> std::allocator<char> > const&, bool
const&, unsigned int
>>>>> const&) ()
>>>>>>>>> from
>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0
>>>>>>>>>>> #2  0x00007fffebbf3d97 in ReadConfigDir(Configuration&,
>>>>>>>>>>> std::__cxx11::basic_string<char, std::char_traits<char>,
>>>>>>>>>>> std::allocator<char> > const&, bool
const&, unsigned int
>>>>> const&) ()
>>>>>>>>> from
>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0
>>>>>>>>>>> #3  0x00007fffebc5e9aa in pkgInitConfig(Configuration&)
>> ()
>>> from
>>>>>>>>>>> /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0
>>>>>>>>>>> #4  0x00007ffff29d5c48 in ?? () from
>>>>>>> /usr/lib/python3/dist-packages/
>>>>>>>>>>> apt_pkg.cpython-35m-x86_64-linux-gnu.so
>>>>>>>>>>> #5  0x00000000004ea10f in PyCFunction_Call ()
>>>>>>>>>>> #6  0x0000000000536d94 in PyEval_EvalFrameEx
()
>>>>>>>>>>> #7  0x000000000053fc97 in ?? ()
>>>>>>>>>>> #8  0x00000000005409bf in PyEval_EvalCode ()
>>>>>>>>>>> #9  0x000000000054a328 in ?? ()
>>>>>>>>>>> #10 0x00000000004ea1c6 in PyCFunction_Call ()
>>>>>>>>>>> #11 0x000000000053d353 in PyEval_EvalFrameEx
()
>>>>>>>>>>> #12 0x000000000053fc97 in ?? ()
>>>>>>>>>>> #13 0x000000000053bc93 in PyEval_EvalFrameEx
()
>>>>>>>>>>> #14 0x000000000053b294 in PyEval_EvalFrameEx
()
>>>>>>>>>>> #15 0x000000000053b294 in PyEval_EvalFrameEx
()
>>>>>>>>>>> #16 0x000000000053b294 in PyEval_EvalFrameEx
()
>>>>>>>>>>> #17 0x0000000000540b0b in PyEval_EvalCodeEx ()
>>>>>>>>>>> #18 0x00000000004ec2e3 in ?? ()
>>>>>>>>>>> #19 0x00000000005c20e7 in PyObject_Call ()
>>>>>>>>>>> 
>>>>>>>>>>> I was using fresh DLAMI ubuntu 18.0 and CUDA
10.0, built
>>> with
>>>>>>>>> USE_CUDA=1,
>>>>>>>>>>> USE_CUDNN=1, the rest are default values.
>>>>>>>>>>> 
>>>>>>>>>>> -Zhi
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Change to +1, I figured out that it was due to the
>>> dependencies. I
>>>>>>> still
>>>>>>>>> have issue using DL base AMI with python3, but I will
not
>>> regard
>>>>> it as
>>>>>>> a
>>>>>>>>> blocker to 1.5 release.
>>>>>>>>> Tested Gluon-CV training and works fine.
>>>>>>>>> 
>>>>>>>>> -Zhi
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> 
>> 

Mime
View raw message