mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Johannes Schulte <johannes.schu...@gmail.com>
Subject Re: "LLR with time"
Date Tue, 14 Nov 2017 20:02:11 GMT
✓

On Mon, Nov 13, 2017 at 3:32 AM, Ted Dunning <ted.dunning@gmail.com> wrote:

> Regarding overfitting, don't forget dithering. That can be the most
> important single step you take in building a good recommender.
>
> Dithering can be inversely proportional to amount of exposures so far if
> you like to give novel items more exposure.
>
> This doesn't have to be very fancy. I have had very good results by
> generating a long list of recommendations, computing a pseudo score based
> on rank, adding a bit of noise and resorting. I also scanned down the list
> and penalized items that showed insufficient diversity.  Then I resorted
> again. Typically, the pseudo score was something like exp(-r) where r is
> rank.
>
> The noise scale is adjusted to leave a good proportion of originally
> recommended items in the first page. It could have easily been scaled by
> 1/sqrt(exposures) to let the newbies move around more.
>
> The parameters here should be adjusted a bit based on experiments, but a
> heuristic first hack works pretty well as a start.
>
>
>
>
>
> On Sun, Nov 12, 2017 at 10:34 PM, Pat Ferrel <pat@occamsmachete.com>
> wrote:
>
> > Part of what Ted is talking about can be seen in the carousels on Netflix
> > or Amazon. Some are not recommendations like “trending” videos, or “new”
> > videos, or “prime” videos (substitute your own promotions here). Nothing
> to
> > do with recommender created items but presented along with
> > recommender-based carousels. They are based on analytics or business
> rules
> > and ideally have some randomness built in. The reason for this is 1) it
> > works by exposing users to items that they would not see in
> recommendations
> > and 2) it provides data to build the recommender model from.
> >
> > A recommender cannot work in an app that has no non-recommended items
> > displayed or there will be no un-biased data to create recommendations
> > from. This would lead to crippling overfitting. Most apps have placements
> > like the ones mentioned above and also have search and browse. However
> you
> > do it, it must be prominent and aways available. The moral of this
> > paragraph is; don’t try to make everything a recommendation, it will be
> > self-defeating. In fact make sure not every video watch comes from a
> > recommendation.
> >
> > Likewise think of placements (reflecting a particular recommender use) as
> > experimentation grounds. Try things like finding a recommended category
> and
> > then recommending items in that category all based on user behavior. Or
> try
> > a placement based on a single thing a user watched like “because you
> > watched xyz you might like these”. Don’t just show the most popular
> > categories for the user and recommend items in them. This would be a type
> > of overfitting too.
> >
> > I’m sure we have strayed far from your original question but maybe it’s
> > covered somewhere in here.
> >
> >
> > On Nov 12, 2017, at 12:11 PM, Johannes Schulte <
> johannes.schulte@gmail.com>
> > wrote:
> >
> > I did "second order" recommendations before but more to fight sparsity
> and
> > find more significant associations in situations with less traffic, so
> > recommending categories instead of products. There needs to be some third
> > order sorting / boosting like you mentioned with "new music", or maybe
> > popularity or hotness to avoid quasi-random order. For events with
> limited
> > lifetime it's probably some mixture of spatial distance and freshness.
> >
> > We will definetely keep an eye on the generation process of data for new
> > items. It depends on the domain but in the time of multi channel
> promotion
> > of videos, shows and products, it's also helps that there is traffic
> driven
> > from external sources.
> >
> > Thanks for the detailed  hints - now it's time to see what comes out of
> > this.
> >
> > Johannes
> >
> > On Sun, Nov 12, 2017 at 7:52 AM, Ted Dunning <ted.dunning@gmail.com>
> > wrote:
> >
> > > Events have the natural good quality that having a cold start means
> that
> > > you will naturally favor recent interactions simply because there won't
> > be
> > > any old interactions to deal with.
> > >
> > > Unfortunately, that also means that you will likely be facing serious
> > cold
> > > start issues all the time. I have used two strategies to deal with cold
> > > starts, both fairly successfully.
> > >
> > > *Method 1: Second order recommendation*
> > >
> > > For novel items with no history, you typically do have some kind of
> > > information about the content. For an event, you may know the
> performer,
> > > the organizer, the venue, possibly something about the content of the
> > event
> > > as well (especially for a tour event). As such, you can build a
> > recommender
> > > that recommends this secondary information and then do a search with
> > > recommended secondary information to find events. This actually works
> > > pretty well, at least for the domains where I have used (music and
> > videos).
> > > For instance, in music, you can easily recommend a new album based on
> the
> > > artist (s) and track list.
> > >
> > > The trick here is to determine when and how to blend in normal
> > > recommendations. One way is query blending where you combine the second
> > > order query with a normal recommendation query, but I think that a fair
> > bit
> > > of experimentation is warranted here.
> > >
> > > *Method 2: What's new and what's trending*
> > >
> > > It is always important to provide alternative avenues of information
> > > gathering for recommendation. Especially for the user generated video
> > case,
> > > there was pretty high interest in the "What's new" and "What's hot"
> > pages.
> > > If you do a decent job of dithering here, you keep reasonably good
> > content
> > > on the what's new page longer than content that doesn't pull. That
> > > maintains interest in the page. Similarly, you can have a bit of a
> lower
> > > bar for new content to be classified as hot than established content.
> > That
> > > way you keep the page fresh (because new stuff appears transiently),
> but
> > > you also have a fair bit of really good stuff as well. If done well,
> > these
> > > pages will provide enough interactions with new items so that they
> don't
> > > start entirely cold. You may need to have genre specific or location
> > > specific versions of these pages to avoid interesting content being
> > > overwhelmed. You might also be able to spot content that has intense
> > > interest from a sub-population as opposed to diffuse interest from a
> mass
> > > population.
> > >
> > > You can also use novelty and trending boosts for content in the normal
> > > recommendation engine. I have avoided this in the past because I felt
> it
> > > was better to have specialized pages for what's new and hot rather than
> > > because I had data saying it was bad to do. I have put a very weak
> > > recommendation effect on the what's hot pages so that people tend to
> see
> > > trending material that they like. That doesn't help on what's new pages
> > for
> > > obvious reasons unless you use a touch of second order recommendation.
> > >
> > >
> > >
> > >
> > >
> > > On Sat, Nov 11, 2017 at 11:00 PM, Johannes Schulte <
> > > johannes.schulte@gmail.com> wrote:
> > >
> > >> Well the greece thing was just an example for a thing you don't know
> > >> upfront - it could be any of the modeled feature on the cross
> > recommender
> > >> input side (user segment, country, city, previous buys), some
> > > subpopulation
> > >> getting active, so the current approach, probably with sampling that
> > >> favours newer events, will be the best here. Luckily a sampling
> strategy
> > > is
> > >> a big topic anyway since we're trying to go for the near real time
> way -
> > >> pat, you talked about it some while ago on this list and i still have
> to
> > >> look at the flink talk from trevor grant but I'm really eager to
> attack
> > >> this after years of batch :)
> > >>
> > >> Thanks for your thoughts, I am happy I can rule something out given
> the
> > >> domain (poisson llr). Luckily the domain I'm working on is event
> > >> recommendations, so there is a natural deterministic item expiry (as
> > >> compared to christmas like stuff).
> > >>
> > >> Again,
> > >> thanks!
> > >>
> > >>
> > >> On Sat, Nov 11, 2017 at 7:00 PM, Ted Dunning <ted.dunning@gmail.com>
> > >> wrote:
> > >>
> > >>> Inline.
> > >>>
> > >>> On Sat, Nov 11, 2017 at 6:31 PM, Pat Ferrel <pat@occamsmachete.com>
> > >> wrote:
> > >>>
> > >>>> If Mahout were to use http://bit.ly/poisson-llr it would tend to
> > > favor
> > >>>> new events in calculating the LLR score for later use in the
> > > threshold
> > >>> for
> > >>>> whether a co or cross-occurrence iss incorporated in the model.
> > >>>
> > >>>
> > >>> I don't think that this would actually help for most recommendation
> > >>> purposes.
> > >>>
> > >>> It might help to determine that some item or other has broken out of
> > >>> historical rates. Thus, we might have "hotness" as a detected feature
> > >> that
> > >>> could be used as a boost at recommendation time. We might also have
> > > "not
> > >>> hotness" as a negative boost feature.
> > >>>
> > >>> Since we have a pretty good handle on the "other" counts, I don't
> think
> > >>> that the Poisson test would help much with the cooccurrence stuff
> > > itself.
> > >>>
> > >>> Changing the sampling rule could make a difference to temporality and
> > >> would
> > >>> be more like what Johannes is asking about.
> > >>>
> > >>>
> > >>>> But it doesn’t relate to popularity as I think Ted is saying.
> > >>>>
> > >>>> Are you looking for 1) personal recommendations biased by hotness
in
> > >>>> Greece or 2) things hot in Greece?
> > >>>>
> > >>>> 1) create a secondary indicator for “watched in some locale”
the
> > >> local-id
> > >>>> uses a country-code+postal-code maybe but not lat-lon. Something
> that
> > >>>> includes a good number of people/events. The the query would be
> > >> user-id,
> > >>>> and user-locale. This would yield personal recs preferred in the
> > > user’s
> > >>>> locale. Athens-west-side in this case.
> > >>>>
> > >>>
> > >>> And this works in the current regime. Simply add location tags to the
> > >> user
> > >>> histories and do cooccurrence against content. Locations will pop out
> > > as
> > >>> indicators for some content and not for others. Then when somebody
> > >> appears
> > >>> in some location, their tags will retrieve localized content.
> > >>>
> > >>> For localization based on strict geography, say for restaurant
> search,
> > > we
> > >>> can just add business rules based on geo-search. A very large bank
> > >> customer
> > >>> of ours does that, for instance.
> > >>>
> > >>>
> > >>>> 2) split the data into locales and do the hot calc I mention. The
> > > query
> > >>>> would have no user-id since it is not personalized but would yield
> > > “hot
> > >>> in
> > >>>> Greece”
> > >>>>
> > >>>
> > >>> I think that this is a good approach.
> > >>>
> > >>>
> > >>>>
> > >>>> Ted’s “Christmas video” tag is what I was calling a business
rule
> and
> > >> can
> > >>>> be added to either of the above techniques.
> > >>>>
> > >>>
> > >>> But the (not) hotness feature might help with automated this.
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>>
> > >>>> On Nov 11, 2017, at 4:01 AM, Ted Dunning <ted.dunning@gmail.com>
> > >> wrote:
> > >>>>
> > >>>> So ... there are a few different threads here.
> > >>>>
> > >>>> 1) LLR but with time. Quite possible, but not really what Johannes
> is
> > >>>> talking about, I think. See http://bit.ly/poisson-llr for a quick
> > >>>> discussion.
> > >>>>
> > >>>> 2) time varying recommendation. As Johannes notes, this can make
use
> > > of
> > >>>> windowed counts. The problem is that rarely accessed items should
> > >>> probably
> > >>>> have longer windows so that we use longer term trends when we have
> > > less
> > >>>> data.
> > >>>>
> > >>>> The good news here is that this some part of this is nearly already
> > > in
> > >>> the
> > >>>> code. The trick is that the down-sampling used in the system can
be
> > >>> adapted
> > >>>> to favor recent events over older ones. That means that if the
> > > meaning
> > >> of
> > >>>> something changes over time, the system will catch on. Likewise,
if
> > >>>> something appears out of nowhere, it will quickly train up. This
> > >> handles
> > >>>> the popular in Greece right now problem.
> > >>>>
> > >>>> But this isn't the whole story of changing recommendations. Another
> > >>> problem
> > >>>> that we commonly face is what I call the christmas music issue.
The
> > >> idea
> > >>> is
> > >>>> that there are lots of recommendations for music that are highly
> > >>> seasonal.
> > >>>> Thus, Bing Crosby fans want to hear White Christmas
> > >>>> <https://www.youtube.com/watch?v=P8Ozdqzjigg> until the day
after
> > >>>> christmas
> > >>>> at which point this becomes a really bad recommendation. To some
> > >> degree,
> > >>>> this can be partially dealt with by using temporal tags as
> > > indicators,
> > >>> but
> > >>>> that doesn't really allow a recommendation to be completely shut
> > > down.
> > >>>>
> > >>>> The only way that I have seen to deal with this in the past is
with
> a
> > >>>> manually designed kill switch. As much as possible, we would tag
the
> > >>>> obviously seasonal content and then add a filter to kill or
> downgrade
> > >>> that
> > >>>> content the moment it went out of fashion.
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Sat, Nov 11, 2017 at 9:43 AM, Johannes Schulte <
> > >>>> johannes.schulte@gmail.com> wrote:
> > >>>>
> > >>>>> Pat, thanks for your help. especially the insights on how you
> > > handle
> > >>> the
> > >>>>> system in production and the tips for multiple acyclic buckets.
> > >>>>> Doing the combination signalls when querying sounds okay but
as you
> > >>> say,
> > >>>>> it's always hard to find the right boosts without setting up
some
> > > ltr
> > >>>>> system. If there would be a way to use the hotness when calculating
> > >> the
> > >>>>> indicators for subpopulations it would be great., especially
for a
> > >>> cross
> > >>>>> recommender.
> > >>>>>
> > >>>>> e.g. people in greece _now_ are viewing this show/product 
whatever
> > >>>>>
> > >>>>> And here the popularity of the recommended item in this
> > > subpopulation
> > >>>> could
> > >>>>> be overrseen when just looking at the overall derivatives of
> > >> activity.
> > >>>>>
> > >>>>> Maybe one could do multiple G-Tests using sliding windows
> > >>>>> * itemA&itemB  vs population (classic)
> > >>>>> * itemA&itemB(t) vs itemA&itemB(t-1)
> > >>>>> ..
> > >>>>>
> > >>>>> and derive multiple indicators per item to be indexed.
> > >>>>>
> > >>>>> But this all relies on discretizing time into buckets and not
> > > looking
> > >>> at
> > >>>>> the distribution of time between events like in presentation
above
> > > -
> > >>>> maybe
> > >>>>> there is  something way smarter
> > >>>>>
> > >>>>> Johannes
> > >>>>>
> > >>>>> On Sat, Nov 11, 2017 at 2:50 AM, Pat Ferrel <pat@occamsmachete.com
> > >>
> > >>>> wrote:
> > >>>>>
> > >>>>>> BTW you should take time buckets that are relatively free
of daily
> > >>>> cycles
> > >>>>>> like 3 day, week, or month buckets for “hot”. This
is to remove
> > >>> cyclical
> > >>>>>> affects from the frequencies as much as possible since
you need 3
> > >>>> buckets
> > >>>>>> to see the change in change, 2 for the change, and 1 for
the event
> > >>>>> volume.
> > >>>>>>
> > >>>>>>
> > >>>>>> On Nov 10, 2017, at 4:12 PM, Pat Ferrel <pat@occamsmachete.com>
> > >>> wrote:
> > >>>>>>
> > >>>>>> So your idea is to find anomalies in event frequencies
to detect
> > >> “hot”
> > >>>>>> items?
> > >>>>>>
> > >>>>>> Interesting, maybe Ted will chime in.
> > >>>>>>
> > >>>>>> What I do is take the frequency, first, and second, derivatives
as
> > >>>>>> measures of popularity, increasing popularity, and increasingly
> > >>>>> increasing
> > >>>>>> popularity. Put another way popular, trending, and hot.
This is
> > >> simple
> > >>>> to
> > >>>>>> do by taking 1, 2, or 3 time buckets and looking at the
number of
> > >>>> events,
> > >>>>>> derivative (difference), and second derivative. Ranking
all items
> > > by
> > >>>>> these
> > >>>>>> value gives various measures of popularity or its increase.
> > >>>>>>
> > >>>>>> If your use is in a recommender you can add a ranking field
to all
> > >>> items
> > >>>>>> and query for “hot” by using the ranking you calculated.
> > >>>>>>
> > >>>>>> If you want to bias recommendations by hotness, query with
user
> > >>> history
> > >>>>>> and boost by your hot field. I suspect the hot field will
tend to
> > >>>>> overwhelm
> > >>>>>> your user history in this case as it would if you used
anomalies
> > > so
> > >>>> you’d
> > >>>>>> also have to normalize the hotness to some range closer
to the one
> > >>>>> created
> > >>>>>> by the user history matching score. I haven’t found a
vey good way
> > >> to
> > >>>> mix
> > >>>>>> these in a model so use hot as a method of backfill if
you cannot
> > >>> return
> > >>>>>> enough recommendations or in places where you may want
to show
> > > just
> > >>> hot
> > >>>>>> items. There are several benefits to this method of using
hot to
> > >> rank
> > >>>> all
> > >>>>>> items including the fact that you can apply business rules
to them
> > >>> just
> > >>>>> as
> > >>>>>> normal recommendations—so you can ask for hot in “electronics”
if
> > >> you
> > >>>>> know
> > >>>>>> categories, or hot "in-stock" items, or ...
> > >>>>>>
> > >>>>>> Still anomaly detection does sound like an interesting
approach.
> > >>>>>>
> > >>>>>>
> > >>>>>> On Nov 10, 2017, at 3:13 PM, Johannes Schulte <
> > >>>>> johannes.schulte@gmail.com>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>> Hi "all",
> > >>>>>>
> > >>>>>> I am wondering what would be the best way to incorporate
event
> > > time
> > >>>>>> information into the calculation of the G-Test.
> > >>>>>>
> > >>>>>> There is a claim here
> > >>>>>> https://de.slideshare.net/tdunning/finding-changes-in-real-data
> > >>>>>>
> > >>>>>> saying "Time aware variant of G-Test is possible"
> > >>>>>>
> > >>>>>> I remember i experimented with exponentially decayed counts
some
> > >> years
> > >>>>> ago
> > >>>>>> and this involved changing the counts to doubles, but I
suspect
> > >> there
> > >>> is
> > >>>>>> some smarter way. What I don't get is the relation to a
data
> > >> structure
> > >>>>> like
> > >>>>>> T-Digest when working with a lot of counts / cells for
every
> > >>> combination
> > >>>>> of
> > >>>>>> items. Keeping a t-digest for every combination seems unfeasible.
> > >>>>>>
> > >>>>>> How would one incorporate event time into recommendations
to
> > > detect
> > >>>>>> "hotness" of certain relations? Glad if someone has an
idea...
> > >>>>>>
> > >>>>>> Cheers,
> > >>>>>>
> > >>>>>> Johannes
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message