mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lance Norskog <>
Subject Re: Order-based evalution of recommenders
Date Sun, 31 Oct 2010 02:51:34 GMT
Yes. One algorithm is this:
Given two recommenders using the same data model, request the same
list of user-item prefs.
The two lists have to have the same symbols, just in a different order.

This order-comparing evaluator runs the algorithm against the two
symbol sequences. So, if they both return the same results in a
similar order, the algorithm evaluates the amount of difference. The
smaller the better.

If I have three "reference" recommenders that have small deltas
against each other, and my recommender has a large delta against all
of them, then my recommender is having problems.

On Fri, Oct 29, 2010 at 4:03 AM, Steven Bourke <> wrote:
> When you say the order do you mean the ranking of the recommendations which
> are returned to the user?
> On Fri, Oct 29, 2010 at 9:57 AM, Lance Norskog <> wrote:
>> I've written an evaluator of recommenders that compares the order of
>> recommendations, rather than the nominal preference values. I'm happy
>> with how well it works now. It is a variant of 'Wilcoxon ranking'.
>> Ranking is unforch N^2 for N recommendations. The "ranking" algorithm
>> is, frankly, baffling but it works.
>> My recommender project uses a different numerical space for
>> preferences and data models, so the existing AbsoluteValue evaluator
>> was useless. This order evaluator requires that two recommendation
>> sequences have the same items, but in different order. If two
>> sequences are almost but not quite the same, the "Sloppy Hamming" and
>> the Wilcoxon Ranking scores correlate well, so that means the Wilcoxon
>> Ranking score is tuned.
>> Sloppy Hamming: V1[N] must match any of V2[N-1], V2[N] or V2[N+1] to
>> create a 'true' at position N. There must be a name for this.
>> --
>> Lance Norskog

Lance Norskog

View raw message