spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Debasish Das <debasish.da...@gmail.com>
Subject Re: MLLib - Thoughts about refactoring Updater for LBFGS?
Date Wed, 26 Feb 2014 01:52:09 GMT
Hi DB,

Could you please point me to your spark PR ?

Thanks.
Deb


On Tue, Feb 25, 2014 at 5:03 PM, DB Tsai <dbtsai@alpinenow.com> wrote:

> Hi Deb, Xiangrui
>
> I just moved the LBFGS code to maven central, and cleaned up the code
> a little bit.
>
> https://github.com/AlpineNow/incubator-spark/commits/dbtsai-LBFGS
>
> After looking at Mallet, the api is pretty simple, and it's probably
> can be easily tested
> based on my PR.
>
> It will be tricky to just benchmark the time of optimizers by
> excluding the parallel gradientSum
> and lossSum computation, and I don't have good approach yet. Let's
> compare the accuracy for the time being.
>
> Thanks.
>
> Sincerely,
>
> DB Tsai
> Machine Learning Engineer
> Alpine Data Labs
> --------------------------------------
> Web: http://alpinenow.com/
>
>
> On Tue, Feb 25, 2014 at 12:07 PM, Debasish Das <debasish.das83@gmail.com>
> wrote:
> > Hi DB,
> >
> > I am considering building on your PR and add Mallet as the dependency so
> > that we can run some basic comparisons test on large scale sparse
> datasets
> > that I have.
> >
> > In the meantime, let's discuss if there are other optimization packages
> > that we should try.
> >
> > My wishlist has bounded bfgs as well and I will add it to the PR.
> >
> > About the PR getting merged to mllib, we can plan that later.
> >
> > Thanks.
> > Deb
> >
> >
> >
> > On Tue, Feb 25, 2014 at 11:36 AM, DB Tsai <dbtsai@alpinenow.com> wrote:
> >
> >> I find some comparison between Mallet vs Fortran version. The result
> >> is closed but not the same.
> >>
> >>
> http://t3827.ai-mallet-development.aitalk.info/help-with-l-bfgs-t3827.html
> >>
> >> Here is LBFGS-B
> >> Cost: 0.6902411220175793
> >> Gradient: -5.453609E-007, -2.858372E-008, -1.369706E-007
> >> Theta: -0.014186210102171406, -0.303521206706629, -0.018132348904129902
> >>
> >> And Mallet LBFGS (Tollerance .000000000000001)
> >> Cost: 0.6902412268833071
> >> Gradient: 0.000117, -4.615523E-005, 0.000114
> >> Theta: -0.013914961040040107, -0.30419883021414335,
> -0.016838481937958744
> >>
> >> So this shows me, that Mallet is close, but Plain ol Gradient Descent
> >> and LBFGS-B are really close.
> >> I see that Mallet also has a "LineOptimizer" and "Evaluator" that I
> >> have yet to explore...
> >>
> >> Sincerely,
> >>
> >> DB Tsai
> >> Machine Learning Engineer
> >> Alpine Data Labs
> >> --------------------------------------
> >> Web: http://alpinenow.com/
> >>
> >>
> >> On Tue, Feb 25, 2014 at 11:16 AM, DB Tsai <dbtsai@alpinenow.com> wrote:
> >> > Hi Deb,
> >> >
> >> > On Tue, Feb 25, 2014 at 7:07 AM, Debasish Das <
> debasish.das83@gmail.com>
> >> wrote:
> >> >> Continuation on last email sent by mistake:
> >> >>
> >> >> Is cpl license is compatible with apache ?
> >> >>
> >> >> http://opensource.org/licenses/cpl1.0.php
> >> >
> >> > Based on what I read here, there is no problem to include CPL code in
> >> > apache project
> >> > as long as the code isn't modified, and we include the maven binary.
> >> > https://www.apache.org/legal/3party.html
> >> >
> >> >> Mallet jars are available on maven. They have hessian based solvers
> >> which
> >> >> looked interesting along with bfgs and cg.
> >> >
> >> > We found that hessian based solvers don't scale as the # of features
> >> grow, and
> >> > we have lots of customers trying to train sparse input. That's our
> >> motivation to
> >> > work on L-BFGS which approximate hessian using just a few vectors.
> >> >
> >> > Just take a look at MALLET, and it does have L-BFGS and its variant
> >> OWL-QN
> >> > which can tackle L1 problem. Since implementing L-BFGS is very
> subtle, I
> >> don't
> >> > know the quality of the mallet implementation. Personally, I
> >> > implemented one based
> >> > on textbook, and not very stable. If MALLET is robust, I'll go for it
> >> > since it has more
> >> > features, and already in maven.
> >> >
> >> >> Note that right now the version is not blas optimized. With jblas or
> >> >> netlib-java discussions that's going on it can be improved. Also it
> >> runs on
> >> >> a single thread which can be improved...so there is scope for further
> >> >> improvements in the code.
> >> >
> >> > I think it will not impact performance even it's not blas optimized
> >> > nor multi-threaded,
> >> > since most of the parallelization is in computing gradientSum and
> >> > lossSum in Spark,
> >> > and the optimizer just takes gradientSum, lossSum, and weights to get
> >> > the newWeights.
> >> >
> >> > As a result, 99.9% of time is in computing gradientSum and lossSum.
> >> > Only small amount
> >> > of time is in optimization.
> >> >
> >> >>
> >> >> Basically Xiangrui, is there a push back on making optimizers part
of
> >> spark
> >> >> mllib ? I am exploring cg and qp solvers for spark mllib as well and
> I
> >> am
> >> >> developing these as part of mllib optimization. I was hoping we
> should
> >> be
> >> >> able to publish mllib as a maven artifact later.
> >> >>
> >> >> Thanks.
> >> >> Deb
> >> >
> >> > Thanks.
> >> >
> >> > Sincerely,
> >> >
> >> > DB Tsai
> >> > Machine Learning Engineer
> >> > Alpine Data Labs
> >> > --------------------------------------
> >> > Web: http://alpinenow.com/
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message