lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Hill <>
Subject RE: SweetSpotSimilarity
Date Thu, 01 Mar 2012 18:40:35 GMT
HI Chris,

 I didn't see your response.  Thanks.
Actually I was recently playing in fooplot , an online plotting tool (one of many), to examine
the various formulas and getting a better handle on what they do.

Thanks for the discussion of 'sweetspot'.  I'm thinking this might help others going forward
who come across sweet spot and wonder what that is all about.
 So sweet spot is the range beyond which things get too "junky".  Interesting!   Now I'll
have to get my head around that idea not for fields like your (product) descriptions, but
actual documents written by users
(aka legal documents).  There ARE ridiculous examples in legal documents -- things like giant
long-running class action lawsuits that when printed are measured in meters.  But maybe the
upper tail would not drop off as fast as your product "description" field example or maybe
sweet spot is not really a sensible idea for body fields which run from very small to occasionally
very large.  It also might be the case that cover letters and e-mails while short might not
be really something to heavily discount.  The lower discount range can be ignored by setting
the min of any sweet spot to 1.  Then one starts to wonder if there is really is any level

It is hard to put it all together, but I do appreciate the fact that all (nearly all?) of
the scoring formula is contained in the class Similarity, but that presents its own interesting
When I get that deep in the code the issue is not simply the shape of the equation, but issues
like how tweaking any parameters effects the overall document scores.  For example, consider
the comments about "steepness" related to length norm.  It talks (some) mathematics of the
equation, but until one spends some time with that equation and understanding where they all
fit together, I doubt it jumps out at most folks what large or smaller values mean for terms
and resulting document scores.

One obvious hard to tease out part of the Similarity API is when each part is called -- the
simplest being index time vs. search time -- there is some clues, but when a coder using any
such interesting override is looking at a method that contains the actual  equation, it is
hard to put it all back together if one has just "spelunked" down all kinds of interesting
"twisty little passages all the same" passed Weight, Scorer and all its friends, passing calls
to deprecated APIs (3.4) to get to an actual formula.  It is also not easy for the API documenter
like you, because obviously, while there is normal place any bit of the equation comes into
the overall scoring formula there really is no guarantee some variation of all the related
classes will call things in the normal manner, So I understand your challenge.  Now in everyone's
defense (and for readers of this discussion), some of the best documentation for a bit larger
picture is the abstract class Similarity even though it contains no formulas.

If I get this all figured out myself, maybe I'll submit a talk "changing document relevancy
for newbies" or "What happens if I pull THIS lever?" :-)

The following is one variation of a plot of computeLengthNorm as shown in fooplot*%28abs%28x-100%29%20%2B%20abs%28x%20-%2050000%29%20-%20%2850000-100%29%29%29%2B%201.0%29%29&y3=&y4=&r0=&r1=&r2=&r3=&r4=&px0=&px1=&px2=&px3=&px4=&py0=&py1=&py2=&py3=&py4=&smin0=0&smin1=0&smin2=0&smin3=0&smin4=0&smax0=2pi&smax1=2pi&smax2=2pi&smax3=2pi&smax4=2pi&thetamin0=0&thetamin1=0&thetamin2=0&thetamin3=0&thetamin4=0&thetamax0=2pi&thetamax1=2pi&thetamax2=2pi&thetamax3=2pi&thetamax4=2pi&ipw=1&ixmin=-50&ixmax=150&iymin=-0.5&iymax=1.5&igx=10&igy=0.25&igl=1&igs=1&iax=0&ila=1&xmin=-50&xmax=150&ymin=-0.5&ymax=1.5

It is hard to say where the best play to place graphs and any such helpful discussion; on-line
or in the source tree.


> -----Original Message-----
> From: Chris Hostetter []
> Sent: Tuesday, February 28, 2012 3:15 PM
> To:
> Subject: RE: SweetSpotSimilarity
> : A picture -- or more precisely a graph -- would be worth a 1000 words.
> fair enough.  I think the reason i never committed one initially was because the formula
in the
> javadocs was trivial to plot in gnuplot...
> gnuplot> min=0
> gnuplot> max=2
> gnuplot> base=1.3
> gnuplot> xoffset=10
> gnuplot> set yrange [0:3]
> gnuplot> set xrange [0:20]
> gnuplot> tf(x)=min+(max-min)/2*(((base**(x-xoffset)-base**-(x-xoffset))/
> gnuplot> (base**(x-xoffset)+base**-(x-xoffset)))+1)
> gnuplot> plot tf(x)
> i'll try to get some graphs commited and linked to from the javadocs that make it more
clear how
> tweaking the settings affect the formula
> : Another problem mentioned in the e-mail thread Chris linked is "people
> : who know the 'sweetspot' of their data.", but I have yet to find a
> : definition of what is meant by "sweetspot", so I couldn't say whether I
> : know my data's sweet spot or not.
> hmmm... sorry, i kind of just always took it s self evident.  i'm not even sure how to
define it ... the
> sweetspot is "the sweetspot" ... the range of good values such that things not in the
sweetspot are
> atypical and "less good"
> To give a practical example: when i was working with product data we found that the sweetspot
> the length of a product name was between 4 and 10 terms.  products with less then 4 terms
in the
> name field were usually junk products (ie: "ram" or "mouse") and products with more then
10 terms in
> the name were usually junk products that had keyword stuffing going on.
> likewise we determined that for fields like the "product description" the sweetspot for
tf matching
> was arround 1-5 (if i remember correctly) ...
> because no one term appeared in a "well written" product description more then 5 times
-- any more
> then that was keyword spamming.
> every catalog of products is going to be different, and every domain is going to be *much*
> (ie: if you search books, or encyclopedia articles then the sweetspots are going to be
much larger)
> : Another question is how the tf_hyper_offset parameter might be
> : considered.  It appears to be the inflexion point of the tanh equation,
> : but what term count might a caller consider centering there ( or
> right ... it's the center of your sweetspot if you use hyperbolicTf, you use the value
that makes sense
> for your data.
> : I also note that the JavaDoc says that the default tf_hyper_base ("the
> : base value to be used in the exponential for the hyperbolic function ")
> : value is e. But checking the code the default is actually 1.3 (less than
> : half e).  Should I file a doc bug?
> I'll fix that (if i remember correctly, "e" is the canonical value typically used in
doing hyperbolics for
> some reason, but for tf purposes made for a curve thta was too steep to be generally
useful by
> default so we changed it as soon as it was committed) ... thanks for pointing out the
doc mistake.
> -Hoss
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message