Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Message-ID: <27542586.1160074282495.JavaMail.root@brutus>
Date: Thu, 5 Oct 2006 11:51:22 -0700 (PDT)
From: "Hoss Man (JIRA)" <jira@apache.org>
To: java-dev@lucene.apache.org
Subject: [jira] Commented: (LUCENE-329) Fuzzy query scoring issues
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

    [ http://issues.apache.org/jira/browse/LUCENE-329?page=comments#action_12440213 ] 
            
Hoss Man commented on LUCENE-329:
---------------------------------

I'm not very familiar with this issue, but a quick review of the patch and the existing comments lead me to believe that commiting it as is wouldn't be the cleanest way to deal with this problem:

1) As Mark mentioned, a "disableCoord" option has been added to BooleanQuery which makes the ExpandedTermsQuery class in this patch unneccessary.  furthermore, the ExpandedTermsQuery class is (in my opinion) broken because it ignores any user specific Similarity completely (instead of just ignoring the coord factor)

2) This patch does not include any test cases.


Furthermore, this patch (assuming it does what it says it does) would fundementally alter the scoring of a FuzzyQuery -- the new scoring may seem like the right thing to do for you and for Mark, but without a larger concensus I'm a little worried that perhaps there are just as many FuzzyQuery users out there who are happy with the current behavior and would consider this change a bug.  So any "fix" for this should either be configurable, or have an overwelming amount of support that the change is the "right" thing to do.

(The main issue seems to be that terms with low idf have a larger impact, and that results in rare mis-spellings getting higher scores -- but assuming "clean data" with no mis-spellings, scoring "rare" terms higher seems like the ideal behavior, correct?)


> Fuzzy query scoring issues
> --------------------------
>
>                 Key: LUCENE-329
>                 URL: http://issues.apache.org/jira/browse/LUCENE-329
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Search
>    Affects Versions: 1.2rc5
>         Environment: Operating System: All
> Platform: All
>            Reporter: Mark Harwood
>         Assigned To: Lucene Developers
>         Attachments: patch.txt
>
>
> Queries which automatically produce multiple terms (wildcard, range, prefix, 
> fuzzy etc)currently suffer from two problems:
> 1) Scores for matching documents are significantly smaller than term queries 
> because of the volume of terms introduced (A match on query Foo~ is 0.1 
> whereas a match on query Foo is 1).
> 2) The rarer forms of expanded terms are favoured over those of more common 
> forms because of the IDF. When using Fuzzy queries for example, rare mis-
> spellings typically appear in results before the more common correct spellings.
> I will attach a patch that corrects the issues identified above by 
> 1) Overriding Similarity.coord to counteract the downplaying of scores 
> introduced by expanding terms.
> 2) Taking the IDF factor of the most common form of expanded terms as the 
> basis of scoring all other expanded terms.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org