The CCO algorithm now supports a couple ways to limit indicators by “quality". The new way
is by the value of LLR. We built a tdigest mechanism to look at the overall density produced
with different thresholds. The higher the threshold, the lower the number of indicators and
the lower the density of the resulting indicator matrix but also the higher the MAP score
(of the full recommender). So MAP seems to increase monotonically until it breaks down.
This didn’t match my understanding of LLR, which is actually a test for noncorrelation.
I was expecting high scores to mean highly likelihood of noncorrelation. So the actual formulation
of the code must be reversing that so the higher the score the higher the likelihood that
noncorrelation is *false* (this is a treated as evidence of correlation)
The next observation is that with high thresholds we get higher MAP scores from the recommender
(expected) but this increases monotonically until it breaks down because there are so few
indicators left. This leads us to the conclusion that MAP is not a good way to set the threshold.
We tried to looking are precision (MAP) vs recall (number of people who get recs) and this
gave ambiguous results with the data we had.
Given my questions about how LLR is actually formulated in Mahout I’m unsure how to convert
it into something like a confidence score or some other way to judge the threshold that would
lead to good way to choose a threshold. Any ideas or illumination about how it’s being calculated
or how to judge the threshold?
Long description of motivation:
LLR thresholds are needed when comparing conversion events to things that have very small
dimensionality so maxIndicatorsPerIItem does not work well. For example a location by state
where there are 50, maxIndicatorsPerItem defaults to 50 so you may end up with 50 very week
indicators. If there are strong indicators in the data, thresholds should be the way to find
them. This might lead to a few per item if the data supports it and this should then be useful.
The question above is how to choose a threshold.
