Mailing-List: contact mahout-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mahout-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of michal.shmueli@gmail.com
 designates 209.85.218.222 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=KB2WJQoWsM4S+q6kgVveAJ6Z9YRcOiJxK6RWtInOddPLLsBJKcgX/P8bVArc9H+Hjt
         kEGDWrywTaXJ4pcco96gk8dn5h3gB/2XYy+BE/WgLKXWkFpY2OZ0omMebX/wgX2R1C/S
         v83GrgoL232TxU6ew9D6jYsyqrQeteU/QUmfw=
MIME-Version: 1.0
In-Reply-To: <e2e029610911050416y2dd8975fp2d988589711ae3b6@mail.gmail.com>
References: <394845f40911040422s4639e475ve574aac8348bf7d2@mail.gmail.com>
	 <e2e029610911040448o4232fb78wc95ce3a09efe5040@mail.gmail.com>
	 <394845f40911042258q353dc885ic99597eaea14d48e@mail.gmail.com>
	 <e2e029610911050247k23afe62bjdf62b5c8d01ae3a0@mail.gmail.com>
	 <394845f40911050406r3dd11ddfu24fe95039f6755fd@mail.gmail.com>
	 <e2e029610911050416y2dd8975fp2d988589711ae3b6@mail.gmail.com>
Date: Thu, 5 Nov 2009 15:08:11 +0200
Message-ID: <394845f40911050508m13c3a4f2teb90d76285df8e5d@mail.gmail.com>
Subject: Re: problems with GenericRecommenderIRStatsEvaluator:
From: michal shmueli <michal.shmueli@gmail.com>
To: mahout-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=0016364167f956c7ee04779f6bc7

--0016364167f956c7ee04779f6bc7
Content-Type: text/plain; charset=ISO-8859-1

The way i envision this is the follow: assume user rates 10 items, this 10
are the correct items. Further assume that for recommendation we use subset
of this 10 items, say 70% (leave us with 30% for test) to build the
similarity, etc. Now, during evaluation, we ask from the recommneder for say
k items, and we check how many from the 3 correct item (the 30% of the
tests) are within the k recommended items.
This solutions ignore the ranking on the different items, however, this
could be also added later.

Does it make sense?

thanks,
Michal

On Thu, Nov 5, 2009 at 2:16 PM, Sean Owen <srowen@gmail.com> wrote:

> It doesn't simulate "training" and "test", that's what I'm saying.
> This concept exists in RecommenderEvaluator, not
> RecommenderIRStatsEvaluator. They're reasonably different things.
>
> In RecommenderIRStatsEvaluator, there is instead a "relevance
> threshold" parameter.
>
> But the final parameter, which you refer to, is something else still.
> It simply controls what percentage of all data to use. It's a simple
> way to use a lot less data to produce a result faster.
>
> You are right that in your 'boolean' data, all preference values are
> effectively 1.0. So passing a 1.0 means that all items are considered
> relevant. That's fine, that's reasonable. While the framework
> typically removes all relevant items from a user for test purposes, it
> will remove only up to "at" items -- that is, if you are evaluating
> precision at 5, it will remove up to 5 items. In this case they are
> effectively randomly chosen since all items are equal.
>
> How would you like it to choose the relevant and not relevant items in
> this case? we can figure out how to do it then.
>
> Sean
>
> On Thu, Nov 5, 2009 at 12:06 PM, michal shmueli
> <michal.shmueli@gmail.com> wrote:
> >    >>  I still don't get why this parameter simulates the "training" and
> > the "test". In addition, since my data is Boolean, ain't it mean that
> anyway
> > what is 1 is relevant ? Is there another way to tell the recommender how
> to
> > chose the training and test sets?
>

--0016364167f956c7ee04779f6bc7--