mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roshani Nagmote <roshaninagmo...@gmail.com>
Subject Re: Release plan - MXNET 1.3
Date Fri, 31 Aug 2018 00:38:14 GMT
Hi all,

I was waiting to run nightly builds to pass on the release branch. So RC0
was a bit delayed.
Now, we have tagged the release candidate! Thanks a lot Sheng for helping
with this!

https://github.com/apache/incubator-mxnet/releases/tag/1.3.0.rc0

Voting process to test RC0 will start after Scala package gets published to
maven repositories.
I had an offline discussion with Naveen and Qing who will be working on
publishing Scala packages. They want to include Scala packages in testing
as well.

I will communicate timelines for voting soon.

Thanks,
Roshani



On Thu, Aug 23, 2018 at 10:24 AM kellen sunderland <
kellen.sunderland@gmail.com> wrote:

> Great news!  Thanks for the efforts Roshani + Sheng.
>
> On Thu, Aug 23, 2018 at 6:58 PM Roshani Nagmote <roshaninagmote2@gmail.com
> >
> wrote:
>
> > Release branch v1.3.x was cut yesterday night. Thanks, @Sheng for helping
> > with this and merging a bunch of PRs. I will be running tests on the
> branch
> > and move forward with the release steps now. :)
> > Thanks,
> > Roshani
> >
> > On Wed, Aug 22, 2018 at 11:12 AM Roshani Nagmote <
> > roshaninagmote2@gmail.com>
> > wrote:
> >
> > > Thanks Patric for reviewing the notes. Updated the doc with MKL-DNN
> > points
> > > you mentioned accordingly.
> > >
> > > Regards,
> > > Roshani
> > >
> > > On Tue, Aug 21, 2018 at 8:03 PM Zhao, Patric <patric.zhao@intel.com>
> > > wrote:
> > >
> > >> Hi Roshani,
> > >>
> > >> Good notes :)
> > >>
> > >> Several items about the performance and MKL-DNN in the below, please
> > help
> > >> take a review.
> > >>
> > >> @Da, Alex, if anything about MKL-DNN is missed, feel free to add.
> > >>
> > >> *Performance improvement
> > >> +Support for dot(dns, csr) = dns and dot(dns, csr.T) = dns on CPU
> > >> https://github.com/apache/incubator-mxnet/pull/11113
> > >> +Performance improvement for Batch Dot on CPU from mshadow
> > >> https://github.com/dmlc/mshadow/pull/342
> > >> -Fix the topk regression issue (#12197)
> > >> This is the bugfix rather than performance improvements
> > >>
> > >>
> > >> *MKL-DNN
> > >> More functionality supports:
> > >> +Support more activation functions, "sigmoid", "tanh", "softrelu"
> > >> https://github.com/apache/incubator-mxnet/pull/10336
> > >>
> > >> Debugging functionality:
> > >> +Result check
> > >> https://github.com/apache/incubator-mxnet/pull/12069
> > >> +Backend switch
> > >> https://github.com/apache/incubator-mxnet/pull/12058
> > >>
> > >> Thanks,
> > >>
> > >> --Patric
> > >>
> > >> > -----Original Message-----
> > >> > From: Roshani Nagmote [mailto:roshaninagmote2@gmail.com]
> > >> > Sent: Wednesday, August 22, 2018 1:53 AM
> > >> > To: dev@mxnet.incubator.apache.org
> > >> > Subject: Re: Release plan - MXNET 1.3
> > >> >
> > >> > Hi,
> > >> >
> > >> > Thank you everyone for helping to clear release blockers. CI tests
> > were
> > >> failing
> > >> > so we delayed RC by some time. But now the tests are passing and we
> > are
> > >> > ready to cut the release branch.
> > >> >
> > >> > I have drafted release notes here:
> > >> >
> https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+%28in
> > >> > cubating%29+1.3.0+Release+Notes
> > >> >
> > >> >
> > >> > Please take a look and update if I have missed anything. I will be
> > >> cutting
> > >> > RC0 tomorrow.
> > >> >
> > >> > Thanks,
> > >> > Roshani
> > >> >
> > >> > On Thu, Aug 16, 2018 at 2:28 PM Roshani Nagmote
> > >> > <roshaninagmote2@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > Sure will do. thanks.
> > >> > >
> > >> > > -Roshani
> > >> > >
> > >> > > On Thu, Aug 16, 2018 at 11:53 AM Afrooze, Sina <
> sina.beh@gmail.com>
> > >> > wrote:
> > >> > >
> > >> > >> Hi Roshani - Can you please make sure that this fix (which is
> > already
> > >> > >> merged to master) is also merged to the stable branch for 1.3.0:
> > >> > >> https://github.com/apache/incubator-mxnet/pull/11493 - Thanks,
> > Sina
> > >> > >>
> > >> > >>
> > >> > >> On 8/16/18, 10:51 AM, "Roshani Nagmote"
> > >> > <roshaninagmote2@gmail.com>
> > >> > >> wrote:
> > >> > >>
> > >> > >>     Hi all,
> > >> > >>
> > >> > >>     Release status:
> > >> > >>
> > >> > >>     Currently, for release 1.3.0 there are a couple of issues
> open
> > >> > >> which needs
> > >> > >>     to be resolved before cutting RC.
> > >> > >>
> > >> > >>     The current date we are looking at for cutting RC0 is
> > >> 08/17(Friday).
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >>     Open issues which need to be looked at before cutting RC:
> > >> > >>
> > >> > >>        1. Topk regression issue
> > >> > >>        <https://github.com/apache/incubator-mxnet/issues/12197>
> -
> > >> > >> #12202 PR
> > >> > >>        with fix <
> > >> https://github.com/apache/incubator-mxnet/pull/12202>
> > >> > >>        2. Excessive memory allocation issue
> > >> > >>        <https://github.com/apache/incubator-mxnet/issues/12116>
> -
> > >> > >> #12184 PR
> > >> > >>        with fix <
> > >> https://github.com/apache/incubator-mxnet/pull/12184>
> > >> > >>        3. Test_io.test_csvIter breaks on CentOS
> > >> > >>        <https://github.com/apache/incubator-mxnet/issues/12139>
> -
> > >> > >> #12189 PR
> > >> > >>        with fix
> > >> > >> <https://github.com/apache/incubator-mxnet/pull/12189>
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >>     @committers, could you please help review these PRs and get
> > them
> > >> > >> merged?
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >>     Thanks,
> > >> > >>
> > >> > >>     Roshani
> > >> > >>
> > >> > >>     On Tue, Aug 14, 2018 at 12:46 PM Roshani Nagmote <
> > >> > >> roshaninagmote2@gmail.com>
> > >> > >>     wrote:
> > >> > >>
> > >> > >>     > Talked to the person who ran resnet50 benchmarks offline.
> > Build
> > >> > >> flag was
> > >> > >>     > not properly set so there was a difference in performance
> > >> > >> numbers observed.
> > >> > >>     > There is no issue caught and he was able to get the same
> > >> results as
> > >> > >>     > mentioned here
> > >> https://mxnet.incubator.apache.org/faq/perf.html
> > >> > >>     > <
> > >> https://mxnet.incubator.apache.org/faq/perf.html#scoring-results>
> > >> > >>     >
> > >> > >>     > We are good here.
> > >> > >>     >
> > >> > >>     > Thanks,
> > >> > >>     > Roshani
> > >> > >>     >
> > >> > >>     > On Mon, Aug 13, 2018 at 4:08 PM Roshani Nagmote <
> > >> > >> roshaninagmote2@gmail.com>
> > >> > >>     > wrote:
> > >> > >>     >
> > >> > >>     >> Hi Dom,
> > >> > >>     >>
> > >> > >>     >> I verified resnet50 run on MXNet master branch. Checked on
> > >> > >> single gpu
> > >> > >>     >> machine. Numbers match. I didn't see any performance
> > >> degradation.
> > >> > >>     >>
> > >> https://mxnet.incubator.apache.org/faq/perf.html#scoring-results
> > >> > >>     >>
> > >> > >>     >> Can you please give me more details on the instance type
> and
> > >> > >> script you
> > >> > >>     >> ran exactly so that I can try to reproduce it again?
> > >> > >>     >>
> > >> > >>     >> Thanks,
> > >> > >>     >> Roshani
> > >> > >>     >>
> > >> > >>     >>
> > >> > >>     >> On Mon, Aug 13, 2018 at 12:31 PM Roshani Nagmote <
> > >> > >>     >> roshaninagmote2@gmail.com> wrote:
> > >> > >>     >>
> > >> > >>     >>> This is not a major feature. I meant other new feature
> > >> > >> requests PR won't
> > >> > >>     >>> be accepted in 1.3 release now.
> > >> > >>     >>> Bug fixes will be accepted. I will be trying to reproduce
> > the
> > >> > >> regression
> > >> > >>     >>> Dom mentioned today. :)
> > >> > >>     >>>
> > >> > >>     >>> Thanks,
> > >> > >>     >>> Roshani
> > >> > >>     >>>
> > >> > >>     >>> On Mon, Aug 13, 2018 at 12:06 PM Naveen Swamy
> > >> > >> <mnnaveen@gmail.com
> > >> > >> >
> > >> > >>     >>> wrote:
> > >> > >>     >>>
> > >> > >>     >>>> Is this is a major feature? This is a regression that
> Dom
> > is
> > >> > >> reporting
> > >> > >>     >>>> wrt
> > >> > >>     >>>> to performance
> > >> > >>     >>>>
> > >> > >>     >>>> On Mon, Aug 13, 2018 at 11:38 AM, Roshani Nagmote <
> > >> > >>     >>>> roshaninagmote2@gmail.com
> > >> > >>     >>>> > wrote:
> > >> > >>     >>>>
> > >> > >>     >>>> > Thanks for reporting this issue Dom.
> > >> > >>     >>>> > 08/10 (Frida)y was the major feature freeze date. We
> > >> won't be
> > >> > >>     >>>> accepting any
> > >> > >>     >>>> > new features now for MXNet 1.3 release.
> > >> > >>     >>>> > RC0 will be cut on 08/17(Friday).
> > >> > >>     >>>> >
> > >> > >>     >>>> > Will be verifying the performance degradation issue
> > >> mentioned.
> > >> > >>     >>>> >
> > >> > >>     >>>> > Thanks,
> > >> > >>     >>>> > Roshani
> > >> > >>     >>>> >
> > >> > >>     >>>> > On Mon, Aug 13, 2018 at 8:45 AM Divakaruni, Dominic
> > >> > >>     >>>> > <ddivakar@amazon.com.invalid> wrote:
> > >> > >>     >>>> >
> > >> > >>     >>>> > > Hi all, We tested resnet50 on MXNet built from
> master
> > >> > >> branch on
> > >> > >>     >>>> Friday
> > >> > >>     >>>> > and
> > >> > >>     >>>> > > were seeing degraded performance on GPU - about 50%
> > >> > >> slower compared
> > >> > >>     >>>> to
> > >> > >>     >>>> > > these values here
> > >> > >> https://mxnet.incubator.apache.org/faq/perf.html.
> > >> > >>     >>>> FWIW
> > >> > >>     >>>> > > this slowdown was seen for both MXNet as well as the
> > TRT
> > >> > >> integrated
> > >> > >>     >>>> > MXNet.
> > >> > >>     >>>> > >
> > >> > >>     >>>> > > Something for you all to verify before or after you
> > cut
> > >> > >> the RC.
> > >> > >>     >>>> > >
> > >> > >>     >>>> > > Thx!
> > >> > >>     >>>> > >
> > >> > >>     >>>> > > On 8/13/18, 4:34 AM, "kellen sunderland" <
> > >> > >>     >>>> kellen.sunderland@gmail.com>
> > >> > >>     >>>> > > wrote:
> > >> > >>     >>>> > >
> > >> > >>     >>>> > >     Hey Roshani,
> > >> > >>     >>>> > >
> > >> > >>     >>>> > >     Has a RC branch already been cut?  If so, a
> quick
> > >> heads
> > >> > >> up that
> > >> > >>     >>>> I
> > >> > >>     >>>> > think
> > >> > >>     >>>> > >     this commit should probably get into RC0 for
> 1.3.
> > >> > >>     >>>> > >
> > >> > >>     >>>> > > https://github.com/apache/incubator-mxnet/commit/
> > >> > >>     >>>> > ee8755a2531b322fec29c9c3d2aa3b8738da41f3
> > >> > >>     >>>> > >
> > >> > >>     >>>> > >     It won't cause issues for users, but from a
> > >> versioning
> > >> > >>     >>>> compatibility
> > >> > >>     >>>> > >     perspective it's probably better that we remove
> > >> these
> > >> > >> functions
> > >> > >>     >>>> in
> > >> > >>     >>>> > this
> > >> > >>     >>>> > >     release. This way we don't have to worry about
> > major
> > >> > >> bumps in
> > >> > >>     >>>> the
> > >> > >>     >>>> > next
> > >> > >>     >>>> > >     release if they're removed.
> > >> > >>     >>>> > >
> > >> > >>     >>>> > >     -Kellen
> > >> > >>     >>>> > >
> > >> > >>     >>>> > >
> > >> > >>     >>>> > >     On Fri, Aug 10, 2018 at 7:24 PM Roshani Nagmote
> <
> > >> > >>     >>>> > > roshaninagmote2@gmail.com>
> > >> > >>     >>>> > >     wrote:
> > >> > >>     >>>> > >
> > >> > >>     >>>> > >     > Thanks Kellen and everyone else for working to
> > get
> > >> > >> TensorRT PR
> > >> > >>     >>>> > > merged!
> > >> > >>     >>>> > >     > @Sina, I will be keeping track of that issue
> and
> > >> fixes
> > >> > >> to get
> > >> > >>     >>>> in
> > >> > >>     >>>> > the
> > >> > >>     >>>> > >     > release.
> > >> > >>     >>>> > >     >
> > >> > >>     >>>> > >     > We are starting code freeze for 1.3 release
> > >> today. A
> > >> > >> release
> > >> > >>     >>>> > > candidate will
> > >> > >>     >>>> > >     > be cut on 08/17.
> > >> > >>     >>>> > >     > Feel free to add any other
> comments/suggestions.
> > >> > >>     >>>> > >     >
> > >> > >>     >>>> > >     > Thanks,
> > >> > >>     >>>> > >     > Roshani
> > >> > >>     >>>> > >     >
> > >> > >>     >>>> > >     > On Fri, Aug 10, 2018 at 5:39 AM kellen
> > sunderland
> > >> <
> > >> > >>     >>>> > >     > kellen.sunderland@gmail.com> wrote:
> > >> > >>     >>>> > >     >
> > >> > >>     >>>> > >     > > All merged and ready to go from my side
> > Roshani
> > >> (the
> > >> > >>     >>>> TensorRT
> > >> > >>     >>>> > PR).
> > >> > >>     >>>> > >     > >
> > >> > >>     >>>> > >     > > I agree with Sina that issue 12116 looks
> it's
> > a
> > >> > >> blocker.
> > >> > >>     >>>> I'll
> > >> > >>     >>>> > try
> > >> > >>     >>>> > > and
> > >> > >>     >>>> > >     > > reproduce it locally to get another
> datapoint.
> > >> > >>     >>>> > >     > >
> > >> > >>     >>>> > >     > > On Fri, Aug 10, 2018 at 3:15 AM Afrooze,
> Sina
> > <
> > >> > >>     >>>> > sina.beh@gmail.com>
> > >> > >>     >>>> > >     > wrote:
> > >> > >>     >>>> > >     > >
> > >> > >>     >>>> > >     > > > Hi Roshani - I think this regression issue
> > is
> > >> a
> > >> > >> release
> > >> > >>     >>>> > blocker:
> > >> > >>     >>>> > >     > > >
> > >> > >> https://github.com/apache/incubator-mxnet/issues/12116
> > >> > >>     >>>> - Sina
> > >> > >>     >>>> > >     > > >
> > >> > >>     >>>> > >     > > >
> > >> > >>     >>>> > >     > > > On 8/8/18, 12:40 PM, "Roshani Nagmote" <
> > >> > >>     >>>> > > roshaninagmote2@gmail.com>
> > >> > >>     >>>> > >     > > wrote:
> > >> > >>     >>>> > >     > > >
> > >> > >>     >>>> > >     > > >     Thanks, Kellen for letting me know.
> > >> > >>     >>>> > >     > > >
> > >> > >>     >>>> > >     > > >     On Wed, Aug 8, 2018 at 12:09 PM kellen
> > >> > >> sunderland <
> > >> > >>     >>>> > >     > > >     kellen.sunderland@gmail.com> wrote:
> > >> > >>     >>>> > >     > > >
> > >> > >>     >>>> > >     > > >     > Hey Roshani, I think it should be
> > ready
> > >> by
> > >> > >> Friday.
> > >> > >>     >>>> > >     > > >     >
> > >> > >>     >>>> > >     > > >     > On Tue, Aug 7, 2018, 10:20 PM
> Roshani
> > >> > >> Nagmote <
> > >> > >>     >>>> > >     > > > roshaninagmote2@gmail.com>
> > >> > >>     >>>> > >     > > >     > wrote:
> > >> > >>     >>>> > >     > > >     >
> > >> > >>     >>>> > >     > > >     > > Thanks Kellen. Yes, we were
> treating
> > >> this
> > >> > >> PR as a
> > >> > >>     >>>> > release
> > >> > >>     >>>> > >     > > blocker.
> > >> > >>     >>>> > >     > > > Do you
> > >> > >>     >>>> > >     > > >     > > have any ETA by which it will be
> > >> completed?
> > >> > >>     >>>> Approximate
> > >> > >>     >>>> > > time
> > >> > >>     >>>> > >     > will
> > >> > >>     >>>> > >     > > > also
> > >> > >>     >>>> > >     > > >     > > work.
> > >> > >>     >>>> > >     > > >     > > @zhi, Thanks for bringing this PR
> > into
> > >> > >> notice. I
> > >> > >>     >>>> will
> > >> > >>     >>>> > > keep a
> > >> > >>     >>>> > >     > > track
> > >> > >>     >>>> > >     > > > of it.
> > >> > >>     >>>> > >     > > >     > >
> > >> > >>     >>>> > >     > > >     > > -Roshani
> > >> > >>     >>>> > >     > > >     > >
> > >> > >>     >>>> > >     > > >     > > On Tue, Aug 7, 2018 at 11:30 AM
> > >> Joshua Z.
> > >> > >> Zhang <
> > >> > >>     >>>> > >     > > > cheungchih@gmail.com>
> > >> > >>     >>>> > >     > > >     > > wrote:
> > >> > >>     >>>> > >     > > >     > >
> > >> > >>     >>>> > >     > > >     > > > I strongly suggest to track this
> > PR
> > >> > >>     >>>> > >     > > >     > > >
> > >> > >>     >>>> https://github.com/apache/incubator-mxnet/pull/11908
> > >> > >>     >>>> > <
> > >> > >>     >>>> > >     > > >     > > >
> > >> > >>     >>>> https://github.com/apache/incubator-mxnet/pull/11908
> > >> > >>     >>>> > >
> > >> > >>     >>>> > > in 1.3
> > >> > >>     >>>> > >     > > > release
> > >> > >>     >>>> > >     > > >     > > > which fixed the usability issue
> > for
> > >> > >> lower end
> > >> > >>     >>>> > machines
> > >> > >>     >>>> > > that
> > >> > >>     >>>> > >     > > > don’t have
> > >> > >>     >>>> > >     > > >     > as
> > >> > >>     >>>> > >     > > >     > > > large shared memory space as ec2
> > >> > >> instances.
> > >> > >>     >>>> > >     > > >     > > >
> > >> > >>     >>>> > >     > > >     > > > Best,
> > >> > >>     >>>> > >     > > >     > > >
> > >> > >>     >>>> > >     > > >     > > > - Zhi
> > >> > >>     >>>> > >     > > >     > > >
> > >> > >>     >>>> > >     > > >     > > > > On Aug 7, 2018, at 9:05 AM,
> > >> Roshani
> > >> > >> Nagmote <
> > >> > >>     >>>> > >     > > >     > roshaninagmote2@gmail.com
> > >> > >>     >>>> > >     > > >     > > >
> > >> > >>     >>>> > >     > > >     > > > wrote:
> > >> > >>     >>>> > >     > > >     > > > >
> > >> > >>     >>>> > >     > > >     > > > > Hi all,
> > >> > >>     >>>> > >     > > >     > > > >
> > >> > >>     >>>> > >     > > >     > > > > Right now, we are delaying
> MXNet
> > >> 1.3
> > >> > >> release
> > >> > >>     >>>> for
> > >> > >>     >>>> > > pending
> > >> > >>     >>>> > >     > > > TensorRT PR
> > >> > >>     >>>> > >     > > >     > (
> > >> > >>     >>>> > >     > > >     > > > > https://github.com/apache/
> > >> > >>     >>>> > incubator-mxnet/pull/11325
> > >> > >>     >>>> > > ).
> > >> > >>     >>>> > >     > > >     > > > >
> > >> > >>     >>>> > >     > > >     > > > > I wanted to ask everyone for
> > their
> > >> > >> opinions
> > >> > >>     >>>> if we
> > >> > >>     >>>> > > should
> > >> > >>     >>>> > >     > > delay
> > >> > >>     >>>> > >     > > > the
> > >> > >>     >>>> > >     > > >     > > > release
> > >> > >>     >>>> > >     > > >     > > > > to get tensorRT integration in
> > or
> > >> we
> > >> > >> should go
> > >> > >>     >>>> > ahead
> > >> > >>     >>>> > > with
> > >> > >>     >>>> > >     > the
> > >> > >>     >>>> > >     > > > release
> > >> > >>     >>>> > >     > > >     > > and
> > >> > >>     >>>> > >     > > >     > > > > include tensorRT in next
> > release.
> > >> > >> Please
> > >> > >>     >>>> provide
> > >> > >>     >>>> > >     > suggestions.
> > >> > >>     >>>> > >     > > >     > > > >
> > >> > >>     >>>> > >     > > >     > > > > Thanks,
> > >> > >>     >>>> > >     > > >     > > > > Roshani
> > >> > >>     >>>> > >     > > >     > > > >
> > >> > >>     >>>> > >     > > >     > > > > On Mon, Aug 6, 2018 at 12:45
> AM
> > >> Hagay
> > >> > >> Lupesko
> > >> > >>     >>>> <
> > >> > >>     >>>> > >     > > > lupesko@gmail.com>
> > >> > >>     >>>> > >     > > >     > > wrote:
> > >> > >>     >>>> > >     > > >     > > > >
> > >> > >>     >>>> > >     > > >     > > > >> Some thoughts: why not keep
> it
> > >> out of
> > >> > >> 1.3,
> > >> > >>     >>>> and
> > >> > >>     >>>> > > merge it
> > >> > >>     >>>> > >     > into
> > >> > >>     >>>> > >     > > > master
> > >> > >>     >>>> > >     > > >     > so
> > >> > >>     >>>> > >     > > >     > > > it
> > >> > >>     >>>> > >     > > >     > > > >> can go out with 1.4 instead?
> > >> > >>     >>>> > >     > > >     > > > >> Pros:
> > >> > >>     >>>> > >     > > >     > > > >> - Reduce quality risks for
> 1.3
> > >> > >>     >>>> > >     > > >     > > > >> - More time to test and get
> > >> feedback
> > >> > >> before
> > >> > >>     >>>> > release
> > >> > >>     >>>> > >     > > >     > > > >> - Avoid further delays in 1.3
> > >> release
> > >> > >> (lots
> > >> > >>     >>>> of
> > >> > >>     >>>> > good
> > >> > >>     >>>> > > stuff
> > >> > >>     >>>> > >     > > > there
> > >> > >>     >>>> > >     > > >     > > already
> > >> > >>     >>>> > >     > > >     > > > for
> > >> > >>     >>>> > >     > > >     > > > >> users)
> > >> > >>     >>>> > >     > > >     > > > >> Cons:
> > >> > >>     >>>> > >     > > >     > > > >> - People will need to get
> > master
> > >> to
> > >> > >>     >>>> experiment
> > >> > >>     >>>> > with
> > >> > >>     >>>> > > TRT
> > >> > >>     >>>> > >     > (not
> > >> > >>     >>>> > >     > > > a major
> > >> > >>     >>>> > >     > > >     > > > issue
> > >> > >>     >>>> > >     > > >     > > > >> IMO)
> > >> > >>     >>>> > >     > > >     > > > >>
> > >> > >>     >>>> > >     > > >     > > > >> Besides, TRT requires a build
> > >> flag
> > >> > >> anyway, so
> > >> > >>     >>>> > MXNet
> > >> > >>     >>>> > > users
> > >> > >>     >>>> > >     > > > consuming
> > >> > >>     >>>> > >     > > >     > > > built
> > >> > >>     >>>> > >     > > >     > > > >> packages (PyPi, Scala) will
> > >> anyway
> > >> > >> not be
> > >> > >>     >>>> able to
> > >> > >>     >>>> > > try it
> > >> > >>     >>>> > >     > out
> > >> > >>     >>>> > >     > > > unless
> > >> > >>     >>>> > >     > > >     > > > >> building from source...
> > >> > >>     >>>> > >     > > >     > > > >>
> > >> > >>     >>>> > >     > > >     > > > >> Thoughts?
> > >> > >>     >>>> > >     > > >     > > > >>
> > >> > >>     >>>> > >     > > >     > > > >> On Sun, Aug 5, 2018 at 10:38
> PM
> > >> > >> Steffen
> > >> > >>     >>>> Rochel <
> > >> > >>     >>>> > >     > > >     > > steffenrochel@gmail.com
> > >> > >>     >>>> > >     > > >     > > > >
> > >> > >>     >>>> > >     > > >     > > > >> wrote:
> > >> > >>     >>>> > >     > > >     > > > >>
> > >> > >>     >>>> > >     > > >     > > > >>> Marek, Kellen, Jun, Da,
> Eric,
> > >> myself
> > >> > >> and a
> > >> > >>     >>>> few
> > >> > >>     >>>> > > other
> > >> > >>     >>>> > >     > people
> > >> > >>     >>>> > >     > > >     > discussed
> > >> > >>     >>>> > >     > > >     > > > >>> offline about TensorRT
> > >> integration
> > >> > >> PR (
> > >> > >>     >>>> > >     > > >     > > > >>>
> > >> > >>     >>>> > >
> https://github.com/apache/incubator-mxnet/pull/11325
> > ).
> > >> > >>     >>>> > >     > We
> > >> > >>     >>>> > >     > > > do
> > >> > >>     >>>> > >     > > >     > agree
> > >> > >>     >>>> > >     > > >     > > > that
> > >> > >>     >>>> > >     > > >     > > > >>> it
> > >> > >>     >>>> > >     > > >     > > > >>> would be good to include the
> > PR
> > >> into
> > >> > >>     >>>> upcoming 1.3
> > >> > >>     >>>> > >     > release,
> > >> > >>     >>>> > >     > > > but are
> > >> > >>     >>>> > >     > > >     > > all
> > >> > >>     >>>> > >     > > >     > > > >>> concerned about the risk
> > >> involved
> > >> > >> and the
> > >> > >>     >>>> > breaking
> > >> > >>     >>>> > > API
> > >> > >>     >>>> > >     > > > change. The
> > >> > >>     >>>> > >     > > >     > > > >>> discussion converged to
> > >> following
> > >> > >> proposal.
> > >> > >>     >>>> (1)
> > >> > >>     >>>> > > change to
> > >> > >>     >>>> > >     > > > contrib
> > >> > >>     >>>> > >     > > >     > PR
> > >> > >>     >>>> > >     > > >     > > > and
> > >> > >>     >>>> > >     > > >     > > > >>> (2) define a different top
> > >> level API
> > >> > >> to
> > >> > >>     >>>> indicate
> > >> > >>     >>>> > > that the
> > >> > >>     >>>> > >     > > > package
> > >> > >>     >>>> > >     > > >     > is
> > >> > >>     >>>> > >     > > >     > > > part
> > >> > >>     >>>> > >     > > >     > > > >>> of contrib and experimental
> > >> (details
> > >> > >> of API
> > >> > >>     >>>> TBD
> > >> > >>     >>>> > > between
> > >> > >>     >>>> > >     > > > Marek,
> > >> > >>     >>>> > >     > > >     > Kellen
> > >> > >>     >>>> > >     > > >     > > > and
> > >> > >>     >>>> > >     > > >     > > > >>> Eric). This change would
> allow
> > >> to
> > >> > >> include
> > >> > >>     >>>> TRT
> > >> > >>     >>>> > > integration
> > >> > >>     >>>> > >     > > > with v1.3
> > >> > >>     >>>> > >     > > >     > > to
> > >> > >>     >>>> > >     > > >     > > > >>> enable users to try TRT with
> > >> MXNet,
> > >> > >>     >>>> minimize the
> > >> > >>     >>>> > > risk and
> > >> > >>     >>>> > >     > > > avoid
> > >> > >>     >>>> > >     > > >     > > > breaking
> > >> > >>     >>>> > >     > > >     > > > >>> API change.
> > >> > >>     >>>> > >     > > >     > > > >>> To accommodate the change
> the
> > >> > >> request is to
> > >> > >>     >>>> delay
> > >> > >>     >>>> > > RC for
> > >> > >>     >>>> > >     > a
> > >> > >>     >>>> > >     > > > few
> > >> > >>     >>>> > >     > > >     > days.
> > >> > >>     >>>> > >     > > >     > > > >>>
> > >> > >>     >>>> > >     > > >     > > > >>> Regards,
> > >> > >>     >>>> > >     > > >     > > > >>> Steffen
> > >> > >>     >>>> > >     > > >     > > > >>>
> > >> > >>     >>>> > >     > > >     > > > >>> On Tue, Jul 31, 2018 at 5:08
> > PM
> > >> > >> Roshani
> > >> > >>     >>>> Nagmote <
> > >> > >>     >>>> > >     > > >     > > > >> roshaninagmote2@gmail.com
> > >> > >>     >>>> > >     > > >     > > > >>>>
> > >> > >>     >>>> > >     > > >     > > > >>> wrote:
> > >> > >>     >>>> > >     > > >     > > > >>>
> > >> > >>     >>>> > >     > > >     > > > >>>> Hi,
> > >> > >>     >>>> > >     > > >     > > > >>>>
> > >> > >>     >>>> > >     > > >     > > > >>>> I have created a wiki for
> > >> tracking
> > >> > >> MXNet
> > >> > >>     >>>> 1.3
> > >> > >>     >>>> > > release
> > >> > >>     >>>> > >     > with
> > >> > >>     >>>> > >     > > > the
> > >> > >>     >>>> > >     > > >     > > > timeline.
> > >> > >>     >>>> > >     > > >     > > > >>>> Please take a look here:
> > >> > >>     >>>> > >     > > >     > > > >>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>
> > >> > >>     >>>> > >     > > >     > > > >>>
> > >> > >>     >>>> > >     > > >     > > > >>
> > >> > >>     >>>> > >     > > >     > > >
> > >> > >>     >>>> > >     > > >     > >
> > >> > >>     >>>> > >     > > >     >
> > >> > >>     >>>> > >     > > >
> > >> > >>     >>>> > >     > >
> > >> > >>     >>>> > >     >
> > >> > >>     >>>> > > https://cwiki.apache.org/confluence/display/MXNET/
> > >> > >>     >>>> > Apache+MXNet+%28incubating%29+1.3.0+Release+Status
> > >> > >>     >>>> > >     > > >     > > > >>>>
> > >> > >>     >>>> > >     > > >     > > > >>>> I am still waiting for
> > >> following 2
> > >> > >> PRs to
> > >> > >>     >>>> get
> > >> > >>     >>>> > > merged:
> > >> > >>     >>>> > >     > > >     > > > >>>> TRT integration:
> > >> > >>     >>>> > >     > > >     > >
> > >> > >>     >>>> https://github.com/apache/incubator-mxnet/pull/11325
> > >> > >>     >>>> > >     > > >     > > > >>>> Gluon RNN:
> > >> > >>     >>>> > >     > > >
> > >> > >> https://github.com/apache/incubator-mxnet/pull/11482
> > >> > >>     >>>> > >     > > >     > > > >>>>
> > >> > >>     >>>> > >     > > >     > > > >>>> *Code freeze date is
> > >> > >> 08/02(Thursday).*
> > >> > >>     >>>> Kindly
> > >> > >>     >>>> > try
> > >> > >>     >>>> > > to
> > >> > >>     >>>> > >     > > > complete
> > >> > >>     >>>> > >     > > >     > > ongoing
> > >> > >>     >>>> > >     > > >     > > > >>> work
> > >> > >>     >>>> > >     > > >     > > > >>>> and get these PRs merged.
> > >> > >>     >>>> > >     > > >     > > > >>>>
> > >> > >>     >>>> > >     > > >     > > > >>>> Thanks,
> > >> > >>     >>>> > >     > > >     > > > >>>> Roshani
> > >> > >>     >>>> > >     > > >     > > > >>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>
> > >> > >>     >>>> > >     > > >     > > > >>>> On Mon, Jul 30, 2018 at
> 1:02
> > PM
> > >> > >> Roshani
> > >> > >>     >>>> Nagmote
> > >> > >>     >>>> > <
> > >> > >>     >>>> > >     > > >     > > > >>> roshaninagmote2@gmail.com
> > >> > >>     >>>> > >     > > >     > > > >>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>> wrote:
> > >> > >>     >>>> > >     > > >     > > > >>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>> Hi all,
> > >> > >>     >>>> > >     > > >     > > > >>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>> Here is an update on MXNet
> > 1.3
> > >> > >> release:
> > >> > >>     >>>> > >     > > >     > > > >>>>> I am still waiting for
> > >> following
> > >> > >> PRs to
> > >> > >>     >>>> get
> > >> > >>     >>>> > > merged:
> > >> > >>     >>>> > >     > > >     > > > >>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>> TRT integration:
> > >> > >>     >>>> > >     > > >     > > > >>
> > >> > >>     >>>> > >
> https://github.com/apache/incubator-mxnet/pull/11325
> > >> > >>     >>>> > >     > > >     > > > >>>>> Gluon RNN:
> > >> > >>     >>>> > >     > > >
> > >> > >> https://github.com/apache/incubator-mxnet/pull/11482
> > >> > >>     >>>> > >     > > >     > > > >>>>> Scala examples:
> > >> > >>     >>>> > >     > > >     > > > >>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>>
> > >> > >>     >>>> > >
> https://github.com/apache/incubator-mxnet/pull/11753
> > >> > >>     >>>> > >     > > >     > > > >>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>>
> > >> > >>     >>>> > >
> https://github.com/apache/incubator-mxnet/pull/11621
> > >> > >>     >>>> > >     > > >     > > > >>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>> *New code freeze date is:
> > >> 08/03*
> > >> > >> Please
> > >> > >>     >>>> try to
> > >> > >>     >>>> > > get
> > >> > >>     >>>> > >     > your
> > >> > >>     >>>> > >     > > > ongoing
> > >> > >>     >>>> > >     > > >     > > PRs
> > >> > >>     >>>> > >     > > >     > > > >>>>> merged by then.
> > >> > >>     >>>> > >     > > >     > > > >>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>> @Pedro, I didn't include
> > your
> > >> PRs
> > >> > >> in
> > >> > >>     >>>> tracking
> > >> > >>     >>>> > > list as
> > >> > >>     >>>> > >     > you
> > >> > >>     >>>> > >     > > > said
> > >> > >>     >>>> > >     > > >     > > those
> > >> > >>     >>>> > >     > > >     > > > >>> are
> > >> > >>     >>>> > >     > > >     > > > >>>>> not critical for now.
> Please
> > >> let
> > >> > >> me know
> > >> > >>     >>>> if
> > >> > >>     >>>> > > those needs
> > >> > >>     >>>> > >     > > to
> > >> > >>     >>>> > >     > > > be
> > >> > >>     >>>> > >     > > >     > > > >> included.
> > >> > >>     >>>> > >     > > >     > > > >>>>>
> > >> > >>     >>>> > >
> https://github.com/apache/incubator-mxnet/pull/11636
> > >> > >>     >>>> > >     > > >     > > > >>>>>
> > >> > >>     >>>> > >
> https://github.com/apache/incubator-mxnet/pull/11562
> > >> > >>     >>>> > >     > > >     > > > >>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>> I also have updated
> project
> > >> > >> proposal
> > >> > >>     >>>> cwiki page
> > >> > >>     >>>> > > to
> > >> > >>     >>>> > >     > update
> > >> > >>     >>>> > >     > > > the
> > >> > >>     >>>> > >     > > >     > > status
> > >> > >>     >>>> > >     > > >     > > > >> of
> > >> > >>     >>>> > >     > > >     > > > >>>>> PRs.
> > >> > >>     >>>> > >     > > >     > > > >>>>> <
> > >> > >>     >>>> > >     > > >     > > > >>>>
> > >> > >>     >>>> > >     > > >     > > > >>>
> > >> > >>     >>>> > >     > > >     > > > >>
> > >> > >>     >>>> > >     > > >     > > >
> > >> > >>     >>>> > >     > > >     > >
> > >> > >>     >>>> > >     > > >     >
> > >> > >>     >>>> > >     > > >
> > >> > >>     >>>> > >     > >
> > >> > >>     >>>> > >     >
> > >> > >>     >>>> > > https://cwiki.apache.org/confluence/display/MXNET/
> > >> > >>     >>>> > Project+Proposals+for+next+MXNet+Release
> > >> > >>     >>>> > >     > > >     > > > >>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>> Please let me know if I am
> > >> missing
> > >> > >>     >>>> something.
> > >> > >>     >>>> > >     > > >     > > > >>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>> Thanks,
> > >> > >>     >>>> > >     > > >     > > > >>>>> Roshani
> > >> > >>     >>>> > >     > > >     > > > >>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>> On Thu, Jul 26, 2018 at
> 1:34
> > >> PM
> > >> > >> Pedro
> > >> > >>     >>>> Larroy <
> > >> > >>     >>>> > >     > > >     > > > >>>>
> pedro.larroy.lists@gmail.com
> > >
> > >> > >>     >>>> > >     > > >     > > > >>>>> wrote:
> > >> > >>     >>>> > >     > > >     > > > >>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>>> I would like to get these
> > PR
> > >> > >> merged:
> > >> > >>     >>>> > >     > > >     > > > >>>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>>>
> > >> > >>     >>>> > >
> https://github.com/apache/incubator-mxnet/pull/11636
> > >> > >>     >>>> > >     > > >     > > > >>>>>>
> > >> > >>     >>>> > >
> https://github.com/apache/incubator-mxnet/pull/11562
> > >> > >>     >>>> > >     > > >     > > > >>>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>>> How much longer until the
> > >> code
> > >> > >> freeze?
> > >> > >>     >>>> > >     > > >     > > > >>>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>>> On Thu, Jul 26, 2018 at
> > 1:44
> > >> AM
> > >> > >> Roshani
> > >> > >>     >>>> > Nagmote
> > >> > >>     >>>> > > <
> > >> > >>     >>>> > >     > > >     > > > >>>>>>
> roshaninagmote2@gmail.com>
> > >> > >>     >>>> > >     > > >     > > > >>>>>> wrote:
> > >> > >>     >>>> > >     > > >     > > > >>>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>>>> Hi all,
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>>>> PRs waiting to be merged
> > >> for 1.3
> > >> > >>     >>>> release:
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>
> > >> > >>     >>>> > >
> https://github.com/apache/incubator-mxnet/pull/11325
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>>>> Are there any other PRs
> > >> waiting
> > >> > >> to get
> > >> > >>     >>>> > merged?
> > >> > >>     >>>> > > Please
> > >> > >>     >>>> > >     > > > let me
> > >> > >>     >>>> > >     > > >     > > know.
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>>>> Release blocker issue:
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>
> > >> > >>     >>>> > >     >
> > >> https://github.com/apache/incubator-mxnet/issues/11853
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>>>> @Marco, @Kellen, Thanks
> > for
> > >> > >> bringing up
> > >> > >>     >>>> the
> > >> > >>     >>>> > > important
> > >> > >>     >>>> > >     > > > topic. I
> > >> > >>     >>>> > >     > > >     > > > >> agree
> > >> > >>     >>>> > >     > > >     > > > >>>>>> with
> > >> > >>     >>>> > >     > > >     > > > >>>>>>> you and we(internal
> Amazon
> > >> team)
> > >> > >> will be
> > >> > >>     >>>> > > working on
> > >> > >>     >>>> > >     > > > fixing the
> > >> > >>     >>>> > >     > > >     > > > >>>> disabled
> > >> > >>     >>>> > >     > > >     > > > >>>>>>> tests.
> > >> > >>     >>>> > >     > > >     > > > >>>>>>> Currently, my colleague,
> > >> Hao Jin
> > >> > >> is
> > >> > >>     >>>> working
> > >> > >>     >>>> > on
> > >> > >>     >>>> > >     > > compiling
> > >> > >>     >>>> > >     > > > the
> > >> > >>     >>>> > >     > > >     > list
> > >> > >>     >>>> > >     > > >     > > > >> of
> > >> > >>     >>>> > >     > > >     > > > >>>>>>> disabled tests and
> leading
> > >> the
> > >> > >> effort
> > >> > >>     >>>> to fix
> > >> > >>     >>>> > > them in
> > >> > >>     >>>> > >     > > the
> > >> > >>     >>>> > >     > > > next
> > >> > >>     >>>> > >     > > >     > few
> > >> > >>     >>>> > >     > > >     > > > >>>> days.
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>>>> Thanks,
> > >> > >>     >>>> > >     > > >     > > > >>>>>>> Roshani
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>>>> On Mon, Jul 23, 2018 at
> > >> 6:39 PM
> > >> > >> kellen
> > >> > >>     >>>> > > sunderland <
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>
> > kellen.sunderland@gmail.com
> > >> >
> > >> > >> wrote:
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>> Thanks again for
> > organizing
> > >> > >> Roshani.  I
> > >> > >>     >>>> > > believe the
> > >> > >>     >>>> > >     > > > TensorRT
> > >> > >>     >>>> > >     > > >     > > > >> work
> > >> > >>     >>>> > >     > > >     > > > >>> is
> > >> > >>     >>>> > >     > > >     > > > >>>>>>> ready
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>> for a merge.  Thanks to
> > >> Marek
> > >> > >> and all
> > >> > >>     >>>> the
> > >> > >>     >>>> > > NVIDIA
> > >> > >>     >>>> > >     > > people
> > >> > >>     >>>> > >     > > > for
> > >> > >>     >>>> > >     > > >     > > > >>>> iterating
> > >> > >>     >>>> > >     > > >     > > > >>>>>> on
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>> it.  If possible could
> a
> > >> > >> committer
> > >> > >>     >>>> review,
> > >> > >>     >>>> > > make sure
> > >> > >>     >>>> > >     > > it
> > >> > >>     >>>> > >     > > > meets
> > >> > >>     >>>> > >     > > >     > > > >>> their
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>> expectations and then
> > >> merge?
> > >> > >> PR is
> > >> > >>     >>>> here:
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>>
> > >> > >>     >>>> > >     >
> > >> https://github.com/apache/incubator-mxnet/pull/11325
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>> To Marco's point.  I'd
> > >> > >> recommend we
> > >> > >>     >>>> review
> > >> > >>     >>>> > > some of
> > >> > >>     >>>> > >     > > those
> > >> > >>     >>>> > >     > > >     > > > >> disabled
> > >> > >>     >>>> > >     > > >     > > > >>>>>> tests
> > >> > >>     >>>> > >     > > >     > > > >>>>>>> and
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>> see how likely they are
> > to
> > >> > >> affect users
> > >> > >>     >>>> > > before we
> > >> > >>     >>>> > >     > cut
> > >> > >>     >>>> > >     > > a
> > >> > >>     >>>> > >     > > >     > release.
> > >> > >>     >>>> > >     > > >     > > > >>>>>> Many of
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>> them are obviously not
> > too
> > >> > >> important
> > >> > >>     >>>> from a
> > >> > >>     >>>> > > user's
> > >> > >>     >>>> > >     > > > point of
> > >> > >>     >>>> > >     > > >     > view
> > >> > >>     >>>> > >     > > >     > > > >>>> (e.g.
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>> downloading a
> > >> sometimes-offline
> > >> > >> image
> > >> > >>     >>>> in a
> > >> > >>     >>>> > > test).
> > >> > >>     >>>> > >     > One
> > >> > >>     >>>> > >     > > > idea
> > >> > >>     >>>> > >     > > >     > > > >> would
> > >> > >>     >>>> > >     > > >     > > > >>> be
> > >> > >>     >>>> > >     > > >     > > > >>>>>> to
> > >> > >>     >>>> > >     > > >     > > > >>>>>>> try
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>> and address as many of
> > the
> > >> > >> customer
> > >> > >>     >>>> > impacting
> > >> > >>     >>>> > > issues
> > >> > >>     >>>> > >     > > as
> > >> > >>     >>>> > >     > > >     > possible
> > >> > >>     >>>> > >     > > >     > > > >>>>>> between
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>> code freeze and the RC0
> > >> vote.
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>> On Mon, Jul 23, 2018 at
> > >> 1:23 PM
> > >> > >> Marco
> > >> > >>     >>>> de
> > >> > >>     >>>> > Abreu
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>> <
> > >> marco.g.abreu@googlemail.com
> > >> > >> .invalid>
> > >> > >>     >>>> > wrote:
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>>> Hello Roshani,
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>>>
> > >> > >>     >>>> > >     > > >     > > > >>>>>>>>> frequent releases are
> > >> good and
> > >> > >> I'm
> > >> > >>     >>>> > > supportive for
> > >> > >>     >>>> > >     > > this
> > >> > >>     >>>> > >

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message