Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of peterlkeegan@gmail.com
 designates 209.85.198.238 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=PYjxMAkIkDOz8auvMs5FAMOM9bmo1YVODT+cag0SyUxlhpOolVToR71vmLXa8a8wSb
         fFOuvAG1v6lvjdvbILYW5ZGrgSZvn2HoTQmRRokKxbj9TRIpLMpG4oHBSLmbLnhoBBEv
         SEyE6ebrXTd8nzs6ovzc5MpLWmS71awwSPQWk=
MIME-Version: 1.0
In-Reply-To: <e994873a0903031142n3436a966w95520a06b23c5e1f@mail.gmail.com>
References: <e994873a0903031142n3436a966w95520a06b23c5e1f@mail.gmail.com>
Date: Mon, 9 Mar 2009 09:05:15 -0500
Message-ID: <e994873a0903090705r1a9aa3bfoe04308dd319cb9ef@mail.gmail.com>
Subject: Re: sloppyFreq question
From: Peter Keegan <peterlkeegan@gmail.com>
To: java-user <java-user@lucene.apache.org>
Content-Type: multipart/alternative; boundary=000e0cd20c56b5488c0464b01f5d

--000e0cd20c56b5488c0464b01f5d
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

The reason I asked about Span scoring is that the behavior changed when I
switched from TermQuery to BoostingTermQuery to take advantage of payloads.

It seems to me that a SpanTermQuery and BoostingTermQuery should behave the
same as TermQuery with respect to term frequency. The 'edit distance' isn't
really relevant for these queries, is it?

For a SpanNearQuery that contains SpanTermQueries, the score for a match on
"the quick brown fox" would be lower than a match on "brown fox" because of
the edit distance (4 vs 2). This seems counter intuitive, too.

Any comments?

Thanks,
Peter


On Tue, Mar 3, 2009 at 2:42 PM, Peter Keegan <peterlkeegan@gmail.com> wrote:

> The DefaultSimilarity class defines sloppyFreq as:
>
> public float sloppyFreq(int distance) {
>   return 1.0f / (distance + 1);
> }
>
> For a 'SpanNearQuery', this reduces the effect of the term frequency on the
> score as the number of terms in the span increases. So, for a simple phrase
> query (using spans), the longer the phrase, the lower the TF. For a simple
> SpanTermQuery, the TF is reduced in half (1.0f / 1 + 1).
>
> I'm just wondering why this is the default behavior. For 'SpanTermQuery',
> I'd expect the TF to reflect the actual number of occurrences of the term.
> For a SpanNearQuery, wouldn't it still be the number of occurrences of the
> whole span, not the number of terms in the span?
>
> Thanks,
> Peter
>

--000e0cd20c56b5488c0464b01f5d--