lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doron Cohen (Commented) (JIRA)" <>
Subject [jira] [Commented] (LUCENE-3821) SloppyPhraseScorer sometimes misses documents that ExactPhraseScorer finds.
Date Fri, 09 Mar 2012 21:32:58 GMT


Doron Cohen commented on LUCENE-3821:

Not understanding really how SloppyPhraseScorer works now, but not trying to add confusion
to the issue, I can't help but think this problem is a variant on LevensteinAutomata... in
fact that was the motivation for the new test, i just stole the testing methodology from there
and applied it to this!

Interesting! I was not aware of this. I went and read some about this automaton, It is relevant.

It seems many things are the same but with a few twists:

* fundamentally we are interleaving the streams from the subscorers into the 'index automaton'
'query automaton' is produced from the user-supplied terms

True. In fact, the current code works hard to decide on the "correct interleaving order" -
while if we had a "Perfect Levenstein Automaton" that took care of the computation we could
just interleave, in the term position order (forget about phrase position and all that) and
let the automaton compute the distance. 

This might capture the difficulty in making the sloppy phrase scorer correct: it started with
the algorithm that was optimized for exact matching, and adopted (hacked?) it for approximate

Instead, starting with a model that fits approximate matching, might be easier and cleaner.
I like that. 

* our 'alphabet' is the terms, and holes from position increment are just an additional symbol.
* just like the LevensteinAutomata case, repeats are problematic because they are different
characteristic vectors
* stacked terms at the same position (index or query) just make the automata more complex
(so they arent just strings)

I'm not suggesting we try to re-use any of that code at all, i don't think it will work. But
I wonder if we can re-use even
some of the math to redefine the problem more formally to figure out what minimal state/lookahead
we need for example...

I agree. I'll think of this.

In the meantime I'll commit this partial fix.
> SloppyPhraseScorer sometimes misses documents that ExactPhraseScorer finds.
> ---------------------------------------------------------------------------
>                 Key: LUCENE-3821
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 3.5, 4.0
>            Reporter: Naomi Dushay
>            Assignee: Doron Cohen
>         Attachments: LUCENE-3821-SloppyDecays.patch, LUCENE-3821.patch, LUCENE-3821.patch,
LUCENE-3821.patch, LUCENE-3821.patch, LUCENE-3821_test.patch, schema.xml, solrconfig-test.xml
> The general bug is a case where a phrase with no slop is found,
> but if you add slop its not.
> I committed a test today (TestSloppyPhraseQuery2) that actually triggers this case,
> jenkins just hasn't had enough time to chew on it.
> ant test -Dtestcase=TestSloppyPhraseQuery2 -Dtests.iter=100 is enough to make it fail
on trunk or 3.x

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message