Mailing-List: contact user-help@mahout.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@mahout.apache.org
Received-SPF: pass (athena.apache.org: domain of goksron@gmail.com designates
 74.125.82.170 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=OrqHMaL5Sv4MPvF8YTSv0Uzn3wRBYOTXoUj4X0MaB99L+iHm+AQ2Hlz6U6FY6DXdo4
         vn9qN1lARXB97gcFb7cPFxs7yPdiI9ao9C8/emV8o/yxVUf7+2zfURi0e/k5CQUabVR3
         zc/gw72j1HVmMTxsNAi7Djv6hcwI/eIdyp0RE=
MIME-Version: 1.0
In-Reply-To: <AANLkTik3Pg78XKDOpxAvMfkgSAo3Mejb+9tp2Kn4oL-v@mail.gmail.com>
References: <AANLkTi=DB-9=AjBbO1MBpbuACu3Z=zUN6PzbwBdUzUOW@mail.gmail.com>
	<AANLkTik3Pg78XKDOpxAvMfkgSAo3Mejb+9tp2Kn4oL-v@mail.gmail.com>
Date: Sat, 30 Oct 2010 19:51:34 -0700
Message-ID: <AANLkTik-mJSmpJ90p5BAv5t5TZ2en4cfzfigV-GWjc7f@mail.gmail.com>
Subject: Re: Order-based evalution of recommenders
From: Lance Norskog <goksron@gmail.com>
To: user@mahout.apache.org
Content-Type: text/plain; charset=UTF-8

Yes. One algorithm is this:
Given two recommenders using the same data model, request the same
list of user-item prefs.
The two lists have to have the same symbols, just in a different order.

This order-comparing evaluator runs the algorithm against the two
symbol sequences. So, if they both return the same results in a
similar order, the algorithm evaluates the amount of difference. The
smaller the better.

If I have three "reference" recommenders that have small deltas
against each other, and my recommender has a large delta against all
of them, then my recommender is having problems.

On Fri, Oct 29, 2010 at 4:03 AM, Steven Bourke <sbourke@gmail.com> wrote:
> When you say the order do you mean the ranking of the recommendations which
> are returned to the user?
>
> On Fri, Oct 29, 2010 at 9:57 AM, Lance Norskog <goksron@gmail.com> wrote:
>
>> I've written an evaluator of recommenders that compares the order of
>> recommendations, rather than the nominal preference values. I'm happy
>> with how well it works now. It is a variant of 'Wilcoxon ranking'.
>> Ranking is unforch N^2 for N recommendations. The "ranking" algorithm
>> is, frankly, baffling but it works.
>>
>> http://comp9.psych.cornell.edu/Darlington/normscor.htm
>>
>> My recommender project uses a different numerical space for
>> preferences and data models, so the existing AbsoluteValue evaluator
>> was useless. This order evaluator requires that two recommendation
>> sequences have the same items, but in different order. If two
>> sequences are almost but not quite the same, the "Sloppy Hamming" and
>> the Wilcoxon Ranking scores correlate well, so that means the Wilcoxon
>> Ranking score is tuned.
>>
>> Sloppy Hamming: V1[N] must match any of V2[N-1], V2[N] or V2[N+1] to
>> create a 'true' at position N. There must be a name for this.
>>
>> --
>> Lance Norskog
>> goksron@gmail.com
>>
>


-- 
Lance Norskog
goksron@gmail.com