Yes. One algorithm is this:
Given two recommenders using the same data model, request the same
list of useritem prefs.
The two lists have to have the same symbols, just in a different order.
This ordercomparing evaluator runs the algorithm against the two
symbol sequences. So, if they both return the same results in a
similar order, the algorithm evaluates the amount of difference. The
smaller the better.
If I have three "reference" recommenders that have small deltas
against each other, and my recommender has a large delta against all
of them, then my recommender is having problems.
On Fri, Oct 29, 2010 at 4:03 AM, Steven Bourke <sbourke@gmail.com> wrote:
> When you say the order do you mean the ranking of the recommendations which
> are returned to the user?
>
> On Fri, Oct 29, 2010 at 9:57 AM, Lance Norskog <goksron@gmail.com> wrote:
>
>> I've written an evaluator of recommenders that compares the order of
>> recommendations, rather than the nominal preference values. I'm happy
>> with how well it works now. It is a variant of 'Wilcoxon ranking'.
>> Ranking is unforch N^2 for N recommendations. The "ranking" algorithm
>> is, frankly, baffling but it works.
>>
>> http://comp9.psych.cornell.edu/Darlington/normscor.htm
>>
>> My recommender project uses a different numerical space for
>> preferences and data models, so the existing AbsoluteValue evaluator
>> was useless. This order evaluator requires that two recommendation
>> sequences have the same items, but in different order. If two
>> sequences are almost but not quite the same, the "Sloppy Hamming" and
>> the Wilcoxon Ranking scores correlate well, so that means the Wilcoxon
>> Ranking score is tuned.
>>
>> Sloppy Hamming: V1[N] must match any of V2[N1], V2[N] or V2[N+1] to
>> create a 'true' at position N. There must be a name for this.
>>
>> 
>> Lance Norskog
>> goksron@gmail.com
>>
>

Lance Norskog
goksron@gmail.com
