mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pedro Larroy <pedro.larroy.li...@gmail.com>
Subject Re: Release plan - MXNET 1.3
Date Fri, 31 Aug 2018 09:38:13 GMT
Hi

The armv7 docker builds are broken due to a problem with dockcross. Shall
we add this to the release notes?  I'm working on a fix. Best would be to
get the fix on the release branch.

https://github.com/apache/incubator-mxnet/issues/12421

Pedro

On Fri, Aug 31, 2018 at 2:38 AM Roshani Nagmote <roshaninagmote2@gmail.com>
wrote:

> Hi all,
>
> I was waiting to run nightly builds to pass on the release branch. So RC0
> was a bit delayed.
> Now, we have tagged the release candidate! Thanks a lot Sheng for helping
> with this!
>
> https://github.com/apache/incubator-mxnet/releases/tag/1.3.0.rc0
>
> Voting process to test RC0 will start after Scala package gets published to
> maven repositories.
> I had an offline discussion with Naveen and Qing who will be working on
> publishing Scala packages. They want to include Scala packages in testing
> as well.
>
> I will communicate timelines for voting soon.
>
> Thanks,
> Roshani
>
>
>
> On Thu, Aug 23, 2018 at 10:24 AM kellen sunderland <
> kellen.sunderland@gmail.com> wrote:
>
> > Great news!  Thanks for the efforts Roshani + Sheng.
> >
> > On Thu, Aug 23, 2018 at 6:58 PM Roshani Nagmote <
> roshaninagmote2@gmail.com
> > >
> > wrote:
> >
> > > Release branch v1.3.x was cut yesterday night. Thanks, @Sheng for
> helping
> > > with this and merging a bunch of PRs. I will be running tests on the
> > branch
> > > and move forward with the release steps now. :)
> > > Thanks,
> > > Roshani
> > >
> > > On Wed, Aug 22, 2018 at 11:12 AM Roshani Nagmote <
> > > roshaninagmote2@gmail.com>
> > > wrote:
> > >
> > > > Thanks Patric for reviewing the notes. Updated the doc with MKL-DNN
> > > points
> > > > you mentioned accordingly.
> > > >
> > > > Regards,
> > > > Roshani
> > > >
> > > > On Tue, Aug 21, 2018 at 8:03 PM Zhao, Patric <patric.zhao@intel.com>
> > > > wrote:
> > > >
> > > >> Hi Roshani,
> > > >>
> > > >> Good notes :)
> > > >>
> > > >> Several items about the performance and MKL-DNN in the below, please
> > > help
> > > >> take a review.
> > > >>
> > > >> @Da, Alex, if anything about MKL-DNN is missed, feel free to add.
> > > >>
> > > >> *Performance improvement
> > > >> +Support for dot(dns, csr) = dns and dot(dns, csr.T) = dns on CPU
> > > >> https://github.com/apache/incubator-mxnet/pull/11113
> > > >> +Performance improvement for Batch Dot on CPU from mshadow
> > > >> https://github.com/dmlc/mshadow/pull/342
> > > >> -Fix the topk regression issue (#12197)
> > > >> This is the bugfix rather than performance improvements
> > > >>
> > > >>
> > > >> *MKL-DNN
> > > >> More functionality supports:
> > > >> +Support more activation functions, "sigmoid", "tanh", "softrelu"
> > > >> https://github.com/apache/incubator-mxnet/pull/10336
> > > >>
> > > >> Debugging functionality:
> > > >> +Result check
> > > >> https://github.com/apache/incubator-mxnet/pull/12069
> > > >> +Backend switch
> > > >> https://github.com/apache/incubator-mxnet/pull/12058
> > > >>
> > > >> Thanks,
> > > >>
> > > >> --Patric
> > > >>
> > > >> > -----Original Message-----
> > > >> > From: Roshani Nagmote [mailto:roshaninagmote2@gmail.com]
> > > >> > Sent: Wednesday, August 22, 2018 1:53 AM
> > > >> > To: dev@mxnet.incubator.apache.org
> > > >> > Subject: Re: Release plan - MXNET 1.3
> > > >> >
> > > >> > Hi,
> > > >> >
> > > >> > Thank you everyone for helping to clear release blockers. CI tests
> > > were
> > > >> failing
> > > >> > so we delayed RC by some time. But now the tests are passing and
> we
> > > are
> > > >> > ready to cut the release branch.
> > > >> >
> > > >> > I have drafted release notes here:
> > > >> >
> > https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+%28in
> > > >> > cubating%29+1.3.0+Release+Notes
> > > >> >
> > > >> >
> > > >> > Please take a look and update if I have missed anything. I will be
> > > >> cutting
> > > >> > RC0 tomorrow.
> > > >> >
> > > >> > Thanks,
> > > >> > Roshani
> > > >> >
> > > >> > On Thu, Aug 16, 2018 at 2:28 PM Roshani Nagmote
> > > >> > <roshaninagmote2@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > Sure will do. thanks.
> > > >> > >
> > > >> > > -Roshani
> > > >> > >
> > > >> > > On Thu, Aug 16, 2018 at 11:53 AM Afrooze, Sina <
> > sina.beh@gmail.com>
> > > >> > wrote:
> > > >> > >
> > > >> > >> Hi Roshani - Can you please make sure that this fix (which is
> > > already
> > > >> > >> merged to master) is also merged to the stable branch for
> 1.3.0:
> > > >> > >> https://github.com/apache/incubator-mxnet/pull/11493 - Thanks,
> > > Sina
> > > >> > >>
> > > >> > >>
> > > >> > >> On 8/16/18, 10:51 AM, "Roshani Nagmote"
> > > >> > <roshaninagmote2@gmail.com>
> > > >> > >> wrote:
> > > >> > >>
> > > >> > >>     Hi all,
> > > >> > >>
> > > >> > >>     Release status:
> > > >> > >>
> > > >> > >>     Currently, for release 1.3.0 there are a couple of issues
> > open
> > > >> > >> which needs
> > > >> > >>     to be resolved before cutting RC.
> > > >> > >>
> > > >> > >>     The current date we are looking at for cutting RC0 is
> > > >> 08/17(Friday).
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> > >>     Open issues which need to be looked at before cutting RC:
> > > >> > >>
> > > >> > >>        1. Topk regression issue
> > > >> > >>        <https://github.com/apache/incubator-mxnet/issues/12197
> >
> > -
> > > >> > >> #12202 PR
> > > >> > >>        with fix <
> > > >> https://github.com/apache/incubator-mxnet/pull/12202>
> > > >> > >>        2. Excessive memory allocation issue
> > > >> > >>        <https://github.com/apache/incubator-mxnet/issues/12116
> >
> > -
> > > >> > >> #12184 PR
> > > >> > >>        with fix <
> > > >> https://github.com/apache/incubator-mxnet/pull/12184>
> > > >> > >>        3. Test_io.test_csvIter breaks on CentOS
> > > >> > >>        <https://github.com/apache/incubator-mxnet/issues/12139
> >
> > -
> > > >> > >> #12189 PR
> > > >> > >>        with fix
> > > >> > >> <https://github.com/apache/incubator-mxnet/pull/12189>
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> > >>     @committers, could you please help review these PRs and get
> > > them
> > > >> > >> merged?
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> > >>     Thanks,
> > > >> > >>
> > > >> > >>     Roshani
> > > >> > >>
> > > >> > >>     On Tue, Aug 14, 2018 at 12:46 PM Roshani Nagmote <
> > > >> > >> roshaninagmote2@gmail.com>
> > > >> > >>     wrote:
> > > >> > >>
> > > >> > >>     > Talked to the person who ran resnet50 benchmarks offline.
> > > Build
> > > >> > >> flag was
> > > >> > >>     > not properly set so there was a difference in performance
> > > >> > >> numbers observed.
> > > >> > >>     > There is no issue caught and he was able to get the same
> > > >> results as
> > > >> > >>     > mentioned here
> > > >> https://mxnet.incubator.apache.org/faq/perf.html
> > > >> > >>     > <
> > > >> https://mxnet.incubator.apache.org/faq/perf.html#scoring-results>
> > > >> > >>     >
> > > >> > >>     > We are good here.
> > > >> > >>     >
> > > >> > >>     > Thanks,
> > > >> > >>     > Roshani
> > > >> > >>     >
> > > >> > >>     > On Mon, Aug 13, 2018 at 4:08 PM Roshani Nagmote <
> > > >> > >> roshaninagmote2@gmail.com>
> > > >> > >>     > wrote:
> > > >> > >>     >
> > > >> > >>     >> Hi Dom,
> > > >> > >>     >>
> > > >> > >>     >> I verified resnet50 run on MXNet master branch. Checked
> on
> > > >> > >> single gpu
> > > >> > >>     >> machine. Numbers match. I didn't see any performance
> > > >> degradation.
> > > >> > >>     >>
> > > >> https://mxnet.incubator.apache.org/faq/perf.html#scoring-results
> > > >> > >>     >>
> > > >> > >>     >> Can you please give me more details on the instance type
> > and
> > > >> > >> script you
> > > >> > >>     >> ran exactly so that I can try to reproduce it again?
> > > >> > >>     >>
> > > >> > >>     >> Thanks,
> > > >> > >>     >> Roshani
> > > >> > >>     >>
> > > >> > >>     >>
> > > >> > >>     >> On Mon, Aug 13, 2018 at 12:31 PM Roshani Nagmote <
> > > >> > >>     >> roshaninagmote2@gmail.com> wrote:
> > > >> > >>     >>
> > > >> > >>     >>> This is not a major feature. I meant other new feature
> > > >> > >> requests PR won't
> > > >> > >>     >>> be accepted in 1.3 release now.
> > > >> > >>     >>> Bug fixes will be accepted. I will be trying to
> reproduce
> > > the
> > > >> > >> regression
> > > >> > >>     >>> Dom mentioned today. :)
> > > >> > >>     >>>
> > > >> > >>     >>> Thanks,
> > > >> > >>     >>> Roshani
> > > >> > >>     >>>
> > > >> > >>     >>> On Mon, Aug 13, 2018 at 12:06 PM Naveen Swamy
> > > >> > >> <mnnaveen@gmail.com
> > > >> > >> >
> > > >> > >>     >>> wrote:
> > > >> > >>     >>>
> > > >> > >>     >>>> Is this is a major feature? This is a regression that
> > Dom
> > > is
> > > >> > >> reporting
> > > >> > >>     >>>> wrt
> > > >> > >>     >>>> to performance
> > > >> > >>     >>>>
> > > >> > >>     >>>> On Mon, Aug 13, 2018 at 11:38 AM, Roshani Nagmote <
> > > >> > >>     >>>> roshaninagmote2@gmail.com
> > > >> > >>     >>>> > wrote:
> > > >> > >>     >>>>
> > > >> > >>     >>>> > Thanks for reporting this issue Dom.
> > > >> > >>     >>>> > 08/10 (Frida)y was the major feature freeze date. We
> > > >> won't be
> > > >> > >>     >>>> accepting any
> > > >> > >>     >>>> > new features now for MXNet 1.3 release.
> > > >> > >>     >>>> > RC0 will be cut on 08/17(Friday).
> > > >> > >>     >>>> >
> > > >> > >>     >>>> > Will be verifying the performance degradation issue
> > > >> mentioned.
> > > >> > >>     >>>> >
> > > >> > >>     >>>> > Thanks,
> > > >> > >>     >>>> > Roshani
> > > >> > >>     >>>> >
> > > >> > >>     >>>> > On Mon, Aug 13, 2018 at 8:45 AM Divakaruni, Dominic
> > > >> > >>     >>>> > <ddivakar@amazon.com.invalid> wrote:
> > > >> > >>     >>>> >
> > > >> > >>     >>>> > > Hi all, We tested resnet50 on MXNet built from
> > master
> > > >> > >> branch on
> > > >> > >>     >>>> Friday
> > > >> > >>     >>>> > and
> > > >> > >>     >>>> > > were seeing degraded performance on GPU - about
> 50%
> > > >> > >> slower compared
> > > >> > >>     >>>> to
> > > >> > >>     >>>> > > these values here
> > > >> > >> https://mxnet.incubator.apache.org/faq/perf.html.
> > > >> > >>     >>>> FWIW
> > > >> > >>     >>>> > > this slowdown was seen for both MXNet as well as
> the
> > > TRT
> > > >> > >> integrated
> > > >> > >>     >>>> > MXNet.
> > > >> > >>     >>>> > >
> > > >> > >>     >>>> > > Something for you all to verify before or after
> you
> > > cut
> > > >> > >> the RC.
> > > >> > >>     >>>> > >
> > > >> > >>     >>>> > > Thx!
> > > >> > >>     >>>> > >
> > > >> > >>     >>>> > > On 8/13/18, 4:34 AM, "kellen sunderland" <
> > > >> > >>     >>>> kellen.sunderland@gmail.com>
> > > >> > >>     >>>> > > wrote:
> > > >> > >>     >>>> > >
> > > >> > >>     >>>> > >     Hey Roshani,
> > > >> > >>     >>>> > >
> > > >> > >>     >>>> > >     Has a RC branch already been cut?  If so, a
> > quick
> > > >> heads
> > > >> > >> up that
> > > >> > >>     >>>> I
> > > >> > >>     >>>> > think
> > > >> > >>     >>>> > >     this commit should probably get into RC0 for
> > 1.3.
> > > >> > >>     >>>> > >
> > > >> > >>     >>>> > > https://github.com/apache/incubator-mxnet/commit/
> > > >> > >>     >>>> > ee8755a2531b322fec29c9c3d2aa3b8738da41f3
> > > >> > >>     >>>> > >
> > > >> > >>     >>>> > >     It won't cause issues for users, but from a
> > > >> versioning
> > > >> > >>     >>>> compatibility
> > > >> > >>     >>>> > >     perspective it's probably better that we
> remove
> > > >> these
> > > >> > >> functions
> > > >> > >>     >>>> in
> > > >> > >>     >>>> > this
> > > >> > >>     >>>> > >     release. This way we don't have to worry about
> > > major
> > > >> > >> bumps in
> > > >> > >>     >>>> the
> > > >> > >>     >>>> > next
> > > >> > >>     >>>> > >     release if they're removed.
> > > >> > >>     >>>> > >
> > > >> > >>     >>>> > >     -Kellen
> > > >> > >>     >>>> > >
> > > >> > >>     >>>> > >
> > > >> > >>     >>>> > >     On Fri, Aug 10, 2018 at 7:24 PM Roshani
> Nagmote
> > <
> > > >> > >>     >>>> > > roshaninagmote2@gmail.com>
> > > >> > >>     >>>> > >     wrote:
> > > >> > >>     >>>> > >
> > > >> > >>     >>>> > >     > Thanks Kellen and everyone else for working
> to
> > > get
> > > >> > >> TensorRT PR
> > > >> > >>     >>>> > > merged!
> > > >> > >>     >>>> > >     > @Sina, I will be keeping track of that issue
> > and
> > > >> fixes
> > > >> > >> to get
> > > >> > >>     >>>> in
> > > >> > >>     >>>> > the
> > > >> > >>     >>>> > >     > release.
> > > >> > >>     >>>> > >     >
> > > >> > >>     >>>> > >     > We are starting code freeze for 1.3 release
> > > >> today. A
> > > >> > >> release
> > > >> > >>     >>>> > > candidate will
> > > >> > >>     >>>> > >     > be cut on 08/17.
> > > >> > >>     >>>> > >     > Feel free to add any other
> > comments/suggestions.
> > > >> > >>     >>>> > >     >
> > > >> > >>     >>>> > >     > Thanks,
> > > >> > >>     >>>> > >     > Roshani
> > > >> > >>     >>>> > >     >
> > > >> > >>     >>>> > >     > On Fri, Aug 10, 2018 at 5:39 AM kellen
> > > sunderland
> > > >> <
> > > >> > >>     >>>> > >     > kellen.sunderland@gmail.com> wrote:
> > > >> > >>     >>>> > >     >
> > > >> > >>     >>>> > >     > > All merged and ready to go from my side
> > > Roshani
> > > >> (the
> > > >> > >>     >>>> TensorRT
> > > >> > >>     >>>> > PR).
> > > >> > >>     >>>> > >     > >
> > > >> > >>     >>>> > >     > > I agree with Sina that issue 12116 looks
> > it's
> > > a
> > > >> > >> blocker.
> > > >> > >>     >>>> I'll
> > > >> > >>     >>>> > try
> > > >> > >>     >>>> > > and
> > > >> > >>     >>>> > >     > > reproduce it locally to get another
> > datapoint.
> > > >> > >>     >>>> > >     > >
> > > >> > >>     >>>> > >     > > On Fri, Aug 10, 2018 at 3:15 AM Afrooze,
> > Sina
> > > <
> > > >> > >>     >>>> > sina.beh@gmail.com>
> > > >> > >>     >>>> > >     > wrote:
> > > >> > >>     >>>> > >     > >
> > > >> > >>     >>>> > >     > > > Hi Roshani - I think this regression
> issue
> > > is
> > > >> a
> > > >> > >> release
> > > >> > >>     >>>> > blocker:
> > > >> > >>     >>>> > >     > > >
> > > >> > >> https://github.com/apache/incubator-mxnet/issues/12116
> > > >> > >>     >>>> - Sina
> > > >> > >>     >>>> > >     > > >
> > > >> > >>     >>>> > >     > > >
> > > >> > >>     >>>> > >     > > > On 8/8/18, 12:40 PM, "Roshani Nagmote" <
> > > >> > >>     >>>> > > roshaninagmote2@gmail.com>
> > > >> > >>     >>>> > >     > > wrote:
> > > >> > >>     >>>> > >     > > >
> > > >> > >>     >>>> > >     > > >     Thanks, Kellen for letting me know.
> > > >> > >>     >>>> > >     > > >
> > > >> > >>     >>>> > >     > > >     On Wed, Aug 8, 2018 at 12:09 PM
> kellen
> > > >> > >> sunderland <
> > > >> > >>     >>>> > >     > > >     kellen.sunderland@gmail.com> wrote:
> > > >> > >>     >>>> > >     > > >
> > > >> > >>     >>>> > >     > > >     > Hey Roshani, I think it should be
> > > ready
> > > >> by
> > > >> > >> Friday.
> > > >> > >>     >>>> > >     > > >     >
> > > >> > >>     >>>> > >     > > >     > On Tue, Aug 7, 2018, 10:20 PM
> > Roshani
> > > >> > >> Nagmote <
> > > >> > >>     >>>> > >     > > > roshaninagmote2@gmail.com>
> > > >> > >>     >>>> > >     > > >     > wrote:
> > > >> > >>     >>>> > >     > > >     >
> > > >> > >>     >>>> > >     > > >     > > Thanks Kellen. Yes, we were
> > treating
> > > >> this
> > > >> > >> PR as a
> > > >> > >>     >>>> > release
> > > >> > >>     >>>> > >     > > blocker.
> > > >> > >>     >>>> > >     > > > Do you
> > > >> > >>     >>>> > >     > > >     > > have any ETA by which it will be
> > > >> completed?
> > > >> > >>     >>>> Approximate
> > > >> > >>     >>>> > > time
> > > >> > >>     >>>> > >     > will
> > > >> > >>     >>>> > >     > > > also
> > > >> > >>     >>>> > >     > > >     > > work.
> > > >> > >>     >>>> > >     > > >     > > @zhi, Thanks for bringing this
> PR
> > > into
> > > >> > >> notice. I
> > > >> > >>     >>>> will
> > > >> > >>     >>>> > > keep a
> > > >> > >>     >>>> > >     > > track
> > > >> > >>     >>>> > >     > > > of it.
> > > >> > >>     >>>> > >     > > >     > >
> > > >> > >>     >>>> > >     > > >     > > -Roshani
> > > >> > >>     >>>> > >     > > >     > >
> > > >> > >>     >>>> > >     > > >     > > On Tue, Aug 7, 2018 at 11:30 AM
> > > >> Joshua Z.
> > > >> > >> Zhang <
> > > >> > >>     >>>> > >     > > > cheungchih@gmail.com>
> > > >> > >>     >>>> > >     > > >     > > wrote:
> > > >> > >>     >>>> > >     > > >     > >
> > > >> > >>     >>>> > >     > > >     > > > I strongly suggest to track
> this
> > > PR
> > > >> > >>     >>>> > >     > > >     > > >
> > > >> > >>     >>>> https://github.com/apache/incubator-mxnet/pull/11908
> > > >> > >>     >>>> > <
> > > >> > >>     >>>> > >     > > >     > > >
> > > >> > >>     >>>> https://github.com/apache/incubator-mxnet/pull/11908
> > > >> > >>     >>>> > >
> > > >> > >>     >>>> > > in 1.3
> > > >> > >>     >>>> > >     > > > release
> > > >> > >>     >>>> > >     > > >     > > > which fixed the usability
> issue
> > > for
> > > >> > >> lower end
> > > >> > >>     >>>> > machines
> > > >> > >>     >>>> > > that
> > > >> > >>     >>>> > >     > > > don’t have
> > > >> > >>     >>>> > >     > > >     > as
> > > >> > >>     >>>> > >     > > >     > > > large shared memory space as
> ec2
> > > >> > >> instances.
> > > >> > >>     >>>> > >     > > >     > > >
> > > >> > >>     >>>> > >     > > >     > > > Best,
> > > >> > >>     >>>> > >     > > >     > > >
> > > >> > >>     >>>> > >     > > >     > > > - Zhi
> > > >> > >>     >>>> > >     > > >     > > >
> > > >> > >>     >>>> > >     > > >     > > > > On Aug 7, 2018, at 9:05 AM,
> > > >> Roshani
> > > >> > >> Nagmote <
> > > >> > >>     >>>> > >     > > >     > roshaninagmote2@gmail.com
> > > >> > >>     >>>> > >     > > >     > > >
> > > >> > >>     >>>> > >     > > >     > > > wrote:
> > > >> > >>     >>>> > >     > > >     > > > >
> > > >> > >>     >>>> > >     > > >     > > > > Hi all,
> > > >> > >>     >>>> > >     > > >     > > > >
> > > >> > >>     >>>> > >     > > >     > > > > Right now, we are delaying
> > MXNet
> > > >> 1.3
> > > >> > >> release
> > > >> > >>     >>>> for
> > > >> > >>     >>>> > > pending
> > > >> > >>     >>>> > >     > > > TensorRT PR
> > > >> > >>     >>>> > >     > > >     > (
> > > >> > >>     >>>> > >     > > >     > > > > https://github.com/apache/
> > > >> > >>     >>>> > incubator-mxnet/pull/11325
> > > >> > >>     >>>> > > ).
> > > >> > >>     >>>> > >     > > >     > > > >
> > > >> > >>     >>>> > >     > > >     > > > > I wanted to ask everyone for
> > > their
> > > >> > >> opinions
> > > >> > >>     >>>> if we
> > > >> > >>     >>>> > > should
> > > >> > >>     >>>> > >     > > delay
> > > >> > >>     >>>> > >     > > > the
> > > >> > >>     >>>> > >     > > >     > > > release
> > > >> > >>     >>>> > >     > > >     > > > > to get tensorRT integration
> in
> > > or
> > > >> we
> > > >> > >> should go
> > > >> > >>     >>>> > ahead
> > > >> > >>     >>>> > > with
> > > >> > >>     >>>> > >     > the
> > > >> > >>     >>>> > >     > > > release
> > > >> > >>     >>>> > >     > > >     > > and
> > > >> > >>     >>>> > >     > > >     > > > > include tensorRT in next
> > > release.
> > > >> > >> Please
> > > >> > >>     >>>> provide
> > > >> > >>     >>>> > >     > suggestions.
> > > >> > >>     >>>> > >     > > >     > > > >
> > > >> > >>     >>>> > >     > > >     > > > > Thanks,
> > > >> > >>     >>>> > >     > > >     > > > > Roshani
> > > >> > >>     >>>> > >     > > >     > > > >
> > > >> > >>     >>>> > >     > > >     > > > > On Mon, Aug 6, 2018 at 12:45
> > AM
> > > >> Hagay
> > > >> > >> Lupesko
> > > >> > >>     >>>> <
> > > >> > >>     >>>> > >     > > > lupesko@gmail.com>
> > > >> > >>     >>>> > >     > > >     > > wrote:
> > > >> > >>     >>>> > >     > > >     > > > >
> > > >> > >>     >>>> > >     > > >     > > > >> Some thoughts: why not keep
> > it
> > > >> out of
> > > >> > >> 1.3,
> > > >> > >>     >>>> and
> > > >> > >>     >>>> > > merge it
> > > >> > >>     >>>> > >     > into
> > > >> > >>     >>>> > >     > > > master
> > > >> > >>     >>>> > >     > > >     > so
> > > >> > >>     >>>> > >     > > >     > > > it
> > > >> > >>     >>>> > >     > > >     > > > >> can go out with 1.4
> instead?
> > > >> > >>     >>>> > >     > > >     > > > >> Pros:
> > > >> > >>     >>>> > >     > > >     > > > >> - Reduce quality risks for
> > 1.3
> > > >> > >>     >>>> > >     > > >     > > > >> - More time to test and get
> > > >> feedback
> > > >> > >> before
> > > >> > >>     >>>> > release
> > > >> > >>     >>>> > >     > > >     > > > >> - Avoid further delays in
> 1.3
> > > >> release
> > > >> > >> (lots
> > > >> > >>     >>>> of
> > > >> > >>     >>>> > good
> > > >> > >>     >>>> > > stuff
> > > >> > >>     >>>> > >     > > > there
> > > >> > >>     >>>> > >     > > >     > > already
> > > >> > >>     >>>> > >     > > >     > > > for
> > > >> > >>     >>>> > >     > > >     > > > >> users)
> > > >> > >>     >>>> > >     > > >     > > > >> Cons:
> > > >> > >>     >>>> > >     > > >     > > > >> - People will need to get
> > > master
> > > >> to
> > > >> > >>     >>>> experiment
> > > >> > >>     >>>> > with
> > > >> > >>     >>>> > > TRT
> > > >> > >>     >>>> > >     > (not
> > > >> > >>     >>>> > >     > > > a major
> > > >> > >>     >>>> > >     > > >     > > > issue
> > > >> > >>     >>>> > >     > > >     > > > >> IMO)
> > > >> > >>     >>>> > >     > > >     > > > >>
> > > >> > >>     >>>> > >     > > >     > > > >> Besides, TRT requires a
> build
> > > >> flag
> > > >> > >> anyway, so
> > > >> > >>     >>>> > MXNet
> > > >> > >>     >>>> > > users
> > > >> > >>     >>>> > >     > > > consuming
> > > >> > >>     >>>> > >     > > >     > > > built
> > > >> > >>     >>>> > >     > > >     > > > >> packages (PyPi, Scala) will
> > > >> anyway
> > > >> > >> not be
> > > >> > >>     >>>> able to
> > > >> > >>     >>>> > > try it
> > > >> > >>     >>>> > >     > out
> > > >> > >>     >>>> > >     > > > unless
> > > >> > >>     >>>> > >     > > >     > > > >> building from source...
> > > >> > >>     >>>> > >     > > >     > > > >>
> > > >> > >>     >>>> > >     > > >     > > > >> Thoughts?
> > > >> > >>     >>>> > >     > > >     > > > >>
> > > >> > >>     >>>> > >     > > >     > > > >> On Sun, Aug 5, 2018 at
> 10:38
> > PM
> > > >> > >> Steffen
> > > >> > >>     >>>> Rochel <
> > > >> > >>     >>>> > >     > > >     > > steffenrochel@gmail.com
> > > >> > >>     >>>> > >     > > >     > > > >
> > > >> > >>     >>>> > >     > > >     > > > >> wrote:
> > > >> > >>     >>>> > >     > > >     > > > >>
> > > >> > >>     >>>> > >     > > >     > > > >>> Marek, Kellen, Jun, Da,
> > Eric,
> > > >> myself
> > > >> > >> and a
> > > >> > >>     >>>> few
> > > >> > >>     >>>> > > other
> > > >> > >>     >>>> > >     > people
> > > >> > >>     >>>> > >     > > >     > discussed
> > > >> > >>     >>>> > >     > > >     > > > >>> offline about TensorRT
> > > >> integration
> > > >> > >> PR (
> > > >> > >>     >>>> > >     > > >     > > > >>>
> > > >> > >>     >>>> > >
> > https://github.com/apache/incubator-mxnet/pull/11325
> > > ).
> > > >> > >>     >>>> > >     > We
> > > >> > >>     >>>> > >     > > > do
> > > >> > >>     >>>> > >     > > >     > agree
> > > >> > >>     >>>> > >     > > >     > > > that
> > > >> > >>     >>>> > >     > > >     > > > >>> it
> > > >> > >>     >>>> > >     > > >     > > > >>> would be good to include
> the
> > > PR
> > > >> into
> > > >> > >>     >>>> upcoming 1.3
> > > >> > >>     >>>> > >     > release,
> > > >> > >>     >>>> > >     > > > but are
> > > >> > >>     >>>> > >     > > >     > > all
> > > >> > >>     >>>> > >     > > >     > > > >>> concerned about the risk
> > > >> involved
> > > >> > >> and the
> > > >> > >>     >>>> > breaking
> > > >> > >>     >>>> > > API
> > > >> > >>     >>>> > >     > > > change. The
> > > >> > >>     >>>> > >     > > >     > > > >>> discussion converged to
> > > >> following
> > > >> > >> proposal.
> > > >> > >>     >>>> (1)
> > > >> > >>     >>>> > > change to
> > > >> > >>     >>>> > >     > > > contrib
> > > >> > >>     >>>> > >     > > >     > PR
> > > >> > >>     >>>> > >     > > >     > > > and
> > > >> > >>     >>>> > >     > > >     > > > >>> (2) define a different top
> > > >> level API
> > > >> > >> to
> > > >> > >>     >>>> indicate
> > > >> > >>     >>>> > > that the
> > > >> > >>     >>>> > >     > > > package
> > > >> > >>     >>>> > >     > > >     > is
> > > >> > >>     >>>> > >     > > >     > > > part
> > > >> > >>     >>>> > >     > > >     > > > >>> of contrib and
> experimental
> > > >> (details
> > > >> > >> of API
> > > >> > >>     >>>> TBD
> > > >> > >>     >>>> > > between
> > > >> > >>     >>>> > >     > > > Marek,
> > > >> > >>     >>>> > >     > > >     > Kellen
> > > >> > >>     >>>> > >     > > >     > > > and
> > > >> > >>     >>>> > >     > > >     > > > >>> Eric). This change would
> > allow
> > > >> to
> > > >> > >> include
> > > >> > >>     >>>> TRT
> > > >> > >>     >>>> > > integration
> > > >> > >>     >>>> > >     > > > with v1.3
> > > >> > >>     >>>> > >     > > >     > > to
> > > >> > >>     >>>> > >     > > >     > > > >>> enable users to try TRT
> with
> > > >> MXNet,
> > > >> > >>     >>>> minimize the
> > > >> > >>     >>>> > > risk and
> > > >> > >>     >>>> > >     > > > avoid
> > > >> > >>     >>>> > >     > > >     > > > breaking
> > > >> > >>     >>>> > >     > > >     > > > >>> API change.
> > > >> > >>     >>>> > >     > > >     > > > >>> To accommodate the change
> > the
> > > >> > >> request is to
> > > >> > >>     >>>> delay
> > > >> > >>     >>>> > > RC for
> > > >> > >>     >>>> > >     > a
> > > >> > >>     >>>> > >     > > > few
> > > >> > >>     >>>> > >     > > >     > days.
> > > >> > >>     >>>> > >     > > >     > > > >>>
> > > >> > >>     >>>> > >     > > >     > > > >>> Regards,
> > > >> > >>     >>>> > >     > > >     > > > >>> Steffen
> > > >> > >>     >>>> > >     > > >     > > > >>>
> > > >> > >>     >>>> > >     > > >     > > > >>> On Tue, Jul 31, 2018 at
> 5:08
> > > PM
> > > >> > >> Roshani
> > > >> > >>     >>>> Nagmote <
> > > >> > >>     >>>> > >     > > >     > > > >> roshaninagmote2@gmail.com
> > > >> > >>     >>>> > >     > > >     > > > >>>>
> > > >> > >>     >>>> > >     > > >     > > > >>> wrote:
> > > >> > >>     >>>> > >     > > >     > > > >>>
> > > >> > >>     >>>> > >     > > >     > > > >>>> Hi,
> > > >> > >>     >>>> > >     > > >     > > > >>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>> I have created a wiki for
> > > >> tracking
> > > >> > >> MXNet
> > > >> > >>     >>>> 1.3
> > > >> > >>     >>>> > > release
> > > >> > >>     >>>> > >     > with
> > > >> > >>     >>>> > >     > > > the
> > > >> > >>     >>>> > >     > > >     > > > timeline.
> > > >> > >>     >>>> > >     > > >     > > > >>>> Please take a look here:
> > > >> > >>     >>>> > >     > > >     > > > >>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>
> > > >> > >>     >>>> > >     > > >     > > > >>
> > > >> > >>     >>>> > >     > > >     > > >
> > > >> > >>     >>>> > >     > > >     > >
> > > >> > >>     >>>> > >     > > >     >
> > > >> > >>     >>>> > >     > > >
> > > >> > >>     >>>> > >     > >
> > > >> > >>     >>>> > >     >
> > > >> > >>     >>>> > >
> https://cwiki.apache.org/confluence/display/MXNET/
> > > >> > >>     >>>> > Apache+MXNet+%28incubating%29+1.3.0+Release+Status
> > > >> > >>     >>>> > >     > > >     > > > >>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>> I am still waiting for
> > > >> following 2
> > > >> > >> PRs to
> > > >> > >>     >>>> get
> > > >> > >>     >>>> > > merged:
> > > >> > >>     >>>> > >     > > >     > > > >>>> TRT integration:
> > > >> > >>     >>>> > >     > > >     > >
> > > >> > >>     >>>> https://github.com/apache/incubator-mxnet/pull/11325
> > > >> > >>     >>>> > >     > > >     > > > >>>> Gluon RNN:
> > > >> > >>     >>>> > >     > > >
> > > >> > >> https://github.com/apache/incubator-mxnet/pull/11482
> > > >> > >>     >>>> > >     > > >     > > > >>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>> *Code freeze date is
> > > >> > >> 08/02(Thursday).*
> > > >> > >>     >>>> Kindly
> > > >> > >>     >>>> > try
> > > >> > >>     >>>> > > to
> > > >> > >>     >>>> > >     > > > complete
> > > >> > >>     >>>> > >     > > >     > > ongoing
> > > >> > >>     >>>> > >     > > >     > > > >>> work
> > > >> > >>     >>>> > >     > > >     > > > >>>> and get these PRs merged.
> > > >> > >>     >>>> > >     > > >     > > > >>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>> Thanks,
> > > >> > >>     >>>> > >     > > >     > > > >>>> Roshani
> > > >> > >>     >>>> > >     > > >     > > > >>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>> On Mon, Jul 30, 2018 at
> > 1:02
> > > PM
> > > >> > >> Roshani
> > > >> > >>     >>>> Nagmote
> > > >> > >>     >>>> > <
> > > >> > >>     >>>> > >     > > >     > > > >>> roshaninagmote2@gmail.com
> > > >> > >>     >>>> > >     > > >     > > > >>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>> wrote:
> > > >> > >>     >>>> > >     > > >     > > > >>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>> Hi all,
> > > >> > >>     >>>> > >     > > >     > > > >>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>> Here is an update on
> MXNet
> > > 1.3
> > > >> > >> release:
> > > >> > >>     >>>> > >     > > >     > > > >>>>> I am still waiting for
> > > >> following
> > > >> > >> PRs to
> > > >> > >>     >>>> get
> > > >> > >>     >>>> > > merged:
> > > >> > >>     >>>> > >     > > >     > > > >>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>> TRT integration:
> > > >> > >>     >>>> > >     > > >     > > > >>
> > > >> > >>     >>>> > >
> > https://github.com/apache/incubator-mxnet/pull/11325
> > > >> > >>     >>>> > >     > > >     > > > >>>>> Gluon RNN:
> > > >> > >>     >>>> > >     > > >
> > > >> > >> https://github.com/apache/incubator-mxnet/pull/11482
> > > >> > >>     >>>> > >     > > >     > > > >>>>> Scala examples:
> > > >> > >>     >>>> > >     > > >     > > > >>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>>
> > > >> > >>     >>>> > >
> > https://github.com/apache/incubator-mxnet/pull/11753
> > > >> > >>     >>>> > >     > > >     > > > >>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>>
> > > >> > >>     >>>> > >
> > https://github.com/apache/incubator-mxnet/pull/11621
> > > >> > >>     >>>> > >     > > >     > > > >>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>> *New code freeze date
> is:
> > > >> 08/03*
> > > >> > >> Please
> > > >> > >>     >>>> try to
> > > >> > >>     >>>> > > get
> > > >> > >>     >>>> > >     > your
> > > >> > >>     >>>> > >     > > > ongoing
> > > >> > >>     >>>> > >     > > >     > > PRs
> > > >> > >>     >>>> > >     > > >     > > > >>>>> merged by then.
> > > >> > >>     >>>> > >     > > >     > > > >>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>> @Pedro, I didn't include
> > > your
> > > >> PRs
> > > >> > >> in
> > > >> > >>     >>>> tracking
> > > >> > >>     >>>> > > list as
> > > >> > >>     >>>> > >     > you
> > > >> > >>     >>>> > >     > > > said
> > > >> > >>     >>>> > >     > > >     > > those
> > > >> > >>     >>>> > >     > > >     > > > >>> are
> > > >> > >>     >>>> > >     > > >     > > > >>>>> not critical for now.
> > Please
> > > >> let
> > > >> > >> me know
> > > >> > >>     >>>> if
> > > >> > >>     >>>> > > those needs
> > > >> > >>     >>>> > >     > > to
> > > >> > >>     >>>> > >     > > > be
> > > >> > >>     >>>> > >     > > >     > > > >> included.
> > > >> > >>     >>>> > >     > > >     > > > >>>>>
> > > >> > >>     >>>> > >
> > https://github.com/apache/incubator-mxnet/pull/11636
> > > >> > >>     >>>> > >     > > >     > > > >>>>>
> > > >> > >>     >>>> > >
> > https://github.com/apache/incubator-mxnet/pull/11562
> > > >> > >>     >>>> > >     > > >     > > > >>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>> I also have updated
> > project
> > > >> > >> proposal
> > > >> > >>     >>>> cwiki page
> > > >> > >>     >>>> > > to
> > > >> > >>     >>>> > >     > update
> > > >> > >>     >>>> > >     > > > the
> > > >> > >>     >>>> > >     > > >     > > status
> > > >> > >>     >>>> > >     > > >     > > > >> of
> > > >> > >>     >>>> > >     > > >     > > > >>>>> PRs.
> > > >> > >>     >>>> > >     > > >     > > > >>>>> <
> > > >> > >>     >>>> > >     > > >     > > > >>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>
> > > >> > >>     >>>> > >     > > >     > > > >>
> > > >> > >>     >>>> > >     > > >     > > >
> > > >> > >>     >>>> > >     > > >     > >
> > > >> > >>     >>>> > >     > > >     >
> > > >> > >>     >>>> > >     > > >
> > > >> > >>     >>>> > >     > >
> > > >> > >>     >>>> > >     >
> > > >> > >>     >>>> > >
> https://cwiki.apache.org/confluence/display/MXNET/
> > > >> > >>     >>>> > Project+Proposals+for+next+MXNet+Release
> > > >> > >>     >>>> > >     > > >     > > > >>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>> Please let me know if I
> am
> > > >> missing
> > > >> > >>     >>>> something.
> > > >> > >>     >>>> > >     > > >     > > > >>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>> Thanks,
> > > >> > >>     >>>> > >     > > >     > > > >>>>> Roshani
> > > >> > >>     >>>> > >     > > >     > > > >>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>> On Thu, Jul 26, 2018 at
> > 1:34
> > > >> PM
> > > >> > >> Pedro
> > > >> > >>     >>>> Larroy <
> > > >> > >>     >>>> > >     > > >     > > > >>>>
> > pedro.larroy.lists@gmail.com
> > > >
> > > >> > >>     >>>> > >     > > >     > > > >>>>> wrote:
> > > >> > >>     >>>> > >     > > >     > > > >>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>>> I would like to get
> these
> > > PR
> > > >> > >> merged:
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>
> > > >> > >>     >>>> > >
> > https://github.com/apache/incubator-mxnet/pull/11636
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>
> > > >> > >>     >>>> > >
> > https://github.com/apache/incubator-mxnet/pull/11562
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>>> How much longer until
> the
> > > >> code
> > > >> > >> freeze?
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>>> On Thu, Jul 26, 2018 at
> > > 1:44
> > > >> AM
> > > >> > >> Roshani
> > > >> > >>     >>>> > Nagmote
> > > >> > >>     >>>> > > <
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>
> > roshaninagmote2@gmail.com>
> > > >> > >>     >>>> > >     > > >     > > > >>>>>> wrote:
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>> Hi all,
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>> PRs waiting to be
> merged
> > > >> for 1.3
> > > >> > >>     >>>> release:
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>>
> > > >> > >>     >>>> > >
> > https://github.com/apache/incubator-mxnet/pull/11325
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>> Are there any other
> PRs
> > > >> waiting
> > > >> > >> to get
> > > >> > >>     >>>> > merged?
> > > >> > >>     >>>> > > Please
> > > >> > >>     >>>> > >     > > > let me
> > > >> > >>     >>>> > >     > > >     > > know.
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>> Release blocker issue:
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>>
> > > >> > >>     >>>> > >     >
> > > >> https://github.com/apache/incubator-mxnet/issues/11853
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>> @Marco, @Kellen,
> Thanks
> > > for
> > > >> > >> bringing up
> > > >> > >>     >>>> the
> > > >> > >>     >>>> > > important
> > > >> > >>     >>>> > >     > > > topic. I
> > > >> > >>     >>>> > >     > > >     > > > >> agree
> > > >> > >>     >>>> > >     > > >     > > > >>>>>> with
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>> you and we(internal
> > Amazon
> > > >> team)
> > > >> > >> will be
> > > >> > >>     >>>> > > working on
> > > >> > >>     >>>> > >     > > > fixing the
> > > >> > >>     >>>> > >     > > >     > > > >>>> disabled
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>> tests.
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>> Currently, my
> colleague,
> > > >> Hao Jin
> > > >> > >> is
> > > >> > >>     >>>> working
> > > >> > >>     >>>> > on
> > > >> > >>     >>>> > >     > > compiling
> > > >> > >>     >>>> > >     > > > the
> > > >> > >>     >>>> > >     > > >     > list
> > > >> > >>     >>>> > >     > > >     > > > >> of
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>> disabled tests and
> > leading
> > > >> the
> > > >> > >> effort
> > > >> > >>     >>>> to fix
> > > >> > >>     >>>> > > them in
> > > >> > >>     >>>> > >     > > the
> > > >> > >>     >>>> > >     > > > next
> > > >> > >>     >>>> > >     > > >     > few
> > > >> > >>     >>>> > >     > > >     > > > >>>> days.
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>> Thanks,
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>> Roshani
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>> On Mon, Jul 23, 2018
> at
> > > >> 6:39 PM
> > > >> > >> kellen
> > > >> > >>     >>>> > > sunderland <
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>>
> > > kellen.sunderland@gmail.com
> > > >> >
> > > >> > >> wrote:
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>>> Thanks again for
> > > organizing
> > > >> > >> Roshani.  I
> > > >> > >>     >>>> > > believe the
> > > >> > >>     >>>> > >     > > > TensorRT
> > > >> > >>     >>>> > >     > > >     > > > >> work
> > > >> > >>     >>>> > >     > > >     > > > >>> is
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>> ready
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>>> for a merge.  Thanks
> to
> > > >> Marek
> > > >> > >> and all
> > > >> > >>     >>>> the
> > > >> > >>     >>>> > > NVIDIA
> > > >> > >>     >>>> > >     > > people
> > > >> > >>     >>>> > >     > > > for
> > > >> > >>     >>>> > >     > > >     > > > >>>> iterating
> > > >> > >>     >>>> > >     > > >     > > > >>>>>> on
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>>> it.  If possible
> could
> > a
> > > >> > >> committer
> > > >> > >>     >>>> review,
> > > >> > >>     >>>> > > make sure
> > > >> > >>     >>>> > >     > > it
> > > >> > >>     >>>> > >     > > > meets
> > > >> > >>     >>>> > >     > > >     > > > >>> their
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>>> expectations and then
> > > >> merge?
> > > >> > >> PR is
> > > >> > >>     >>>> here:
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>>>
> > > >> > >>     >>>> > >     >
> > > >> https://github.com/apache/incubator-mxnet/pull/11325
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>>>
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>>> To Marco's point.
> I'd
> > > >> > >> recommend we
> > > >> > >>     >>>> review
> > > >> > >>     >>>> > > some of
> > > >> > >>     >>>> > >     > > those
> > > >> > >>     >>>> > >     > > >     > > > >> disabled
> > > >> > >>     >>>> > >     > > >     > > > >>>>>> tests
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>> and
> > > >> > >>     >>>> > >     > > >     > > > >>>>>>>> see how likely they
> are
> > > to
> > > >> > >> affect users
> > > >> >

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message