mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sandeep krishnamurthy <sandeep.krishn...@gmail.com>
Subject Re: Design Proposal: SVRG Optimization in MXNet Python Module
Date Tue, 21 Aug 2018 19:45:59 GMT
Hello Stephanie,

Welcome to the Apache MXNet community!

Thanks for sharing the design document. Design and proposal looks great to
me. SVRG will definitely be very useful in certain problems with an
objective of coming up with an vector that is close to optimum.
Following are my notes based on our offline discussion:
1. Probably it will be good idea to contribute this SVRGModule in contrib
to start with, unless community thinks otherwise.
2. It will be very useful if we can break the execution into phases -
Single Machine (CPU, 1-GPU, multi-GPU), multi-machine support followed by a
tutorial. Because, I see there are some complexities involved in gradient
aggregation and updates. This will also enable for smaller PRs and
incrementally allow community to try it out.

Process we generally follow in MXNet for new contributions is simple:
1. Design proposal in cwiki and discussion on dev@. (Added you to cwiki and
the proposal review is already initiated. Thanks)
2. JIRA issue creation for the tasks (I can help you with that)
3. PRs (Community will review/contribute and participate. One of the
committer will help you through the review process, I can volunteer to work
with you)

Thanks for your contributions to MXNet.

Best,
Sandeep

On Mon, Aug 20, 2018 at 9:02 PM Stephanie Yuan <stephanieyuan1994@gmail.com>
wrote:

> Hi MXNet dev community,
>
> My name is Stephanie Yuan and it's great to join the MXNet dev family!  I'm
> proposing a new design doc for implementing SVRG optimization technique in
> MXNet Python Module.
>
> *Problem Description: *
> SVRG optimization is a technique that complements SGD, which was first
> proposed in the paper  Accelerating Stochastic Gradient Descent using
> Predicative Variance Reduction
> <
> https://papers.nips.cc/paper/4937-accelerating-stochastic-gradient-descent-using-predictive-variance-reduction.pdf
> >
> in
> 2013.  It has provable guarantees for strongly convex functions and
> converges much faster than SGD. An initial set of experiments using
> YearPredictionMSD dataset has been conducted and yields promising results,
> which is one of the motivations for this proposal.
>
> *Expected Deliverables:*
> The goal is to implement a MXNet Python Module that implements SVRG
> optimization technique.
>
> Detailed implementation approaches and Benchmark results can be found in
> the Confluence design doc
> <
> https://cwiki.apache.org/confluence/display/MXNET/SVRG+Optimization+in+MXNet+Python+Module
> >
> .
>
> Please let me know if you have any questions! Thank you very much for your
> time and considerations!
>
> Cheers,
> Stephanie Yuan
>


-- 
Sandeep Krishnamurthy

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message