Mailing-List: contact mahout-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mahout-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of srowen@gmail.com designates
 209.85.220.224 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Q60p07wKxDKMLybGdoaMe+5OZ2+2FoYn2DZeqaEVVOwWdoWBuEZJy5nXN/6fq50Khn
         HTOnoJPZQNtggETCKY4p4ocAn8sovyo+OC645ADO6K5fVh7ro+DLMn+iByylL5EYVFVm
         BXZtK1IIzfWgK6ycvVCYxHuLuMD8B9Uk1DrIs=
MIME-Version: 1.0
In-Reply-To: <26540395.post@talk.nabble.com>
References: <26530825.post@talk.nabble.com>
	 <e2e029610911260915t50cb8aa7hcc30133838846daa@mail.gmail.com>
	 <26533265.post@talk.nabble.com> <26535849.post@talk.nabble.com>
	 <e2e029610911261809k66d5aa6cv187da85282132584@mail.gmail.com>
	 <26540395.post@talk.nabble.com>
Date: Fri, 27 Nov 2009 12:01:04 +0000
Message-ID: <e2e029610911270401m41464ee2obae237a62157ec58@mail.gmail.com>
Subject: Re: Mahout/Taste covariance between two items
From: Sean Owen <srowen@gmail.com>
To: mahout-user@lucene.apache.org
Content-Type: text/plain; charset=UTF-8

I'm not so familiar with this formula but you seem to be missing
something in the denominator... it's got to normalize somehow. I think
I said divide by standard deviation but that's not quite it. What you
are really summing are the products of z-scores.  See
http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient

But I think you should just use the formulation given in the code?
should be the same result. At least I hope these aren't different
definitions of Pearson!

On Fri, Nov 27, 2009 at 10:20 AM, jamborta <jamborta@gmail.com> wrote:
>
> thanks you. much clearer now.
>
> so for my purpose this will do:
>
> sumXY/N-1
>
> given that the data is 'centered'?
>
>
> On Fri, Nov 27, 2009 at 1:41 AM, jamborta <jamborta@gmail.com> wrote:
>>
>> hi. I tried to figure out how you calcualte pearson correlation, but it
>> looks
>> like you use this formula:
>>
>> sumXY / sqrt(sumX2 * sumY2)
>
> Yes that's right -- this is what Pearson reduces to when the mean of X
> and Y are 0. And they are here -- the implementation 'centers' the
> data.
>
>> where sumXY = sumXY - meanY * sumX;
>> sumX2 = sumX2 - meanX * sumX;
>> sumY2 = sumY2 - meanY * sumY;
>
> You see the lines commented out there? Those are the full forms of the
> expressions, which may make more sense. This is centering the data,
> making the mean 0.
>
> This is a simplification based on the observation that, for example,
> sumX * meanY = sumY * meanX = n * meanY * meanX.
>
>>
>> i don't really understand how you got these equations. could you explain
>> it
>> to me? I thought pearson correlation would be like this
>>
>> E(x_i-meanX)(y_i-meanY) / sdX*sdY
>
> That's right that's the expression for a population correlation, but
> we can really only compute a sample Pearson correlation coefficient,
> yes:
>
>
>> for my project I would need to get sample correlation coefficient which
>> would be something like this:
>>
>> sum(x_i-meanX)(y_i-meanY)/(N-1)
>
> Yeah that's fine too, this is another way of expressing the formula,
> though you're missing the two standard deviations in the denominator.
> It'll be clearer if I note that the mean of X and Y are 0.
>
>
>
> --
> View this message in context: http://old.nabble.com/Mahout-Taste-covariance-between-two-items-tp26530825p26540395.html
> Sent from the Mahout User List mailing list archive at Nabble.com.
>
>