mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frank Scholten <fr...@frankscholten.nl>
Subject Re: Logistic Regression cost function
Date Tue, 14 Jan 2014 09:20:01 GMT
I see the update rule to the beta matrix is derived from pseudo code in the
innermost loop in 'Algorithm 1: Stochastic Gradient Descent' in
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.177.3514&rep=rep1&type=pdf

In the paper the learning rate is put before the gradient of the error
function and the instance value multiplication is put before the gradient.

Perhaps we can rearrange the code like this so it matches with the paper
and add a comment? The only difference then is the perTermAnnealingRate.

        // See 'Algorithm 1: SGD' in
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.177.3514&rep=rep1&type=pdf
        double newValue = beta.getQuick(i, j) + learningRate *
perTermLearningRate(j) * instance.get(j) * gradientBase;

Cheers,

Frank







On Mon, Jan 13, 2014 at 10:54 PM, Frank Scholten <frank@frankscholten.nl>wrote:

> Thanks guys, I have some reading to do :-)
>
>
> On Mon, Jan 13, 2014 at 10:45 PM, Ted Dunning <ted.dunning@gmail.com>wrote:
>
>> The reference is to the web site in general.
>>
>> If anything, this blog is closest:
>>
>> http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.177.3514&rep=rep1&type=pdf
>>
>>
>> On Mon, Jan 13, 2014 at 1:14 PM, Suneel Marthi <suneel_marthi@yahoo.com
>> >wrote:
>>
>> > I think this is the one. Yes, I don't see this paper referenced in the
>> > code sorry about that.
>> > http://leon.bottou.org/publications/pdf/compstat-2010.pdf
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Monday, January 13, 2014 3:51 PM, Frank Scholten <
>> > frank@frankscholten.nl> wrote:
>> >
>> > Do you know which paper it is? He has quite a few publications. I don't
>> see
>> > any mention of one of his papers in the code. I only see
>> > www.eecs.tufts.edu/~dsculley/papers/combined-ranking-and-regression.pdfin
>> > MixedGradient but this is something different.
>> >
>> >
>> >
>> >
>> > On Mon, Jan 13, 2014 at 1:27 PM, Suneel Marthi <suneel_marthi@yahoo.com
>> > >wrote:
>> >
>> > > Mahout's impl is based off of Leon Bottou's paper on this subject.  I
>> > > don't gave the link handy but it's referenced in the code or try
>> google
>> > > search
>> > >
>> > > Sent from my iPhone
>> > >
>> > > > On Jan 13, 2014, at 7:14 AM, Frank Scholten <frank@frankscholten.nl
>> >
>> > > wrote:
>> > > >
>> > > > Hi,
>> > > >
>> > > > I followed the Coursera Machine Learning course quite a while ago
>> and I
>> > > am
>> > > > trying to find out how Mahout implements the Logistic Regression
>> cost
>> > > > function in the code surrounding AbstractOnlineLogisticRegression.
>> > > >
>> > > > I am looking at the train method in AbstractOnlineLogisticRegression
>> > and
>> > > I
>> > > > see online gradient descent step where the beta matrix is updated
>> but
>> > to
>> > > me
>> > > > its unclear how matches with the cost function described at:
>> > > > http://www.holehouse.org/mlclass/06_Logistic_Regression.html
>> > > >
>> > > > Perhaps Mahout uses an optimized approach for that does not directly
>> > map
>> > > > into the formula at that link?
>> > > >
>> > > > Cheers,
>> > > >
>> > > > Frank
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message