Mailing-List: contact dev-help@mahout.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@mahout.apache.org
From: Pat Ferrel <pat@occamsmachete.com>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_DCFE56E2-3C27-4E56-9220-A3D82BFA1A90"
Mime-Version: 1.0 (Mac OS X Mail 10.2 \(3259\))
Subject: LLR thresholds
Message-Id: <9F739984-1F02-4212-B061-3084D9B904BA@occamsmachete.com>
Date: Wed, 8 Mar 2017 08:18:48 -0800
Cc: Ted Dunning <ted.dunning@gmail.com>,
 Ted Dunning <tdunning@maprtech.com>,
 ssc@apache.org
To: Mahout Dev List <dev@mahout.apache.org>
archived-at: Wed, 08 Mar 2017 16:24:42 -0000

--Apple-Mail=_DCFE56E2-3C27-4E56-9220-A3D82BFA1A90
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

The CCO algorithm now supports a couple ways to limit indicators by =
=E2=80=9Cquality". The new way is by the value of LLR. We built a =
t-digest mechanism to look at the overall density produced with =
different thresholds. The higher the threshold, the lower the number of =
indicators and the lower the density of the resulting indicator matrix =
but also the higher the MAP score (of the full recommender). So MAP =
seems to increase monotonically until it breaks down.

This didn=E2=80=99t match my understanding of LLR, which is actually a =
test for non-correlation. I was expecting high scores to mean highly =
likelihood of non-correlation. So the actual formulation of the code =
must be reversing that so the higher the score the higher the likelihood =
that non-correlation is *false* (this is a treated as evidence of =
correlation)

The next observation is that with high thresholds we get higher MAP =
scores from the recommender (expected) but this increases monotonically =
until it breaks down because there are so few indicators left. This =
leads us to the conclusion that MAP is not a good way to set the =
threshold. We tried to looking are precision (MAP) vs recall (number of =
people who get recs) and this gave ambiguous results with the data we =
had.

Given my questions about how LLR is actually formulated in Mahout I=E2=80=99=
m unsure how to convert it into something like a confidence score or =
some other way to judge the threshold that would lead to good way to =
choose a threshold. Any ideas or illumination about how it=E2=80=99s =
being calculated or how to judge the threshold?


Long description of motivation:

LLR thresholds are needed when comparing conversion events to things =
that have very small dimensionality so maxIndicatorsPerIItem does not =
work well. For example a location by state where there are 50, =
maxIndicatorsPerItem defaults to 50 so you may end up with 50 very week =
indicators. If there are strong indicators in the data, thresholds =
should be the way to find them. This might lead to a few per item if the =
data supports it and this should then be useful. The question above is =
how to choose a threshold.=

--Apple-Mail=_DCFE56E2-3C27-4E56-9220-A3D82BFA1A90--