predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pat Ferrel <...@occamsmachete.com>
Subject Re: How to score the model? (lift)
Date Mon, 26 Sep 2016 23:21:16 GMT
A/B testing it the typical way to compare the performance of 2 or more versions of an app.
You should make sure that the versions only vary in 1 way to isolate what you are comparing.
So channel some traffic to a page “without recommendations” and some amount of traffic
to a page “with recommendations” and measure your conversion KPI for both cases. If your
conversion is a sale or purchase you may have to split all users permanently over the test
period. if you are only measuring CTR you may be able to split impressions since you don’t
have to track users as a group all the way to a purchase.
  
On Sep 26, 2016, at 5:45 AM, Gustavo Frederico <gustavo.frederico@thinkwrap.com> wrote:

Allow me to elaborate some more: there's a second question related to
the original one. How to score a model in Production?

Thanks

On Fri, Sep 23, 2016 at 2:58 PM, Gustavo Frederico
<gustavo.frederico@thinkwrap.com> wrote:
> Considering PIO + the UR template, what are some of the ways to score
> the model? I can think of some A/B testing but that would imply
> relying on things outside the PIO + UR stack. Is there a way to
> measure lift or "score the model"? Cross-validation with separate
> testing set?
> 
> Thanks
> 
> Gustavo


Mime
View raw message