Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.co.uk;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding;
  b=G3NpSKGukzubmSumEn33GccE5YNL+MtgGumd9FfLg6Vuvx3Z0/IN2i5kDTvVFllqXIZjJR1d60+3vZxs6KlxlYmp3pmuyQ/15JSGyJ86OKKo0CAfwFmIqtq1eWswydgVOXS4irynaEm2KmimG4W6evZ6IA5DCUK15KKcwCLr3gc=;
Message-ID: <457524.14998.qm@web24716.mail.ird.yahoo.com>
Date: Thu, 27 Aug 2009 14:25:10 +0000 (GMT)
From: Mark Harwood <markharw00d@yahoo.co.uk>
Subject: Re: FuzzyLikeThis query and exact matches
To: "java-dev@lucene.apache.org" <java-dev@lucene.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

I think those boosts shown are reflecting the edit distance. What we can't =
see from this is that the Similarity class used in execution is using the s=
ame IDF for all terms. The other factors at play will be the term frequency=
 in the doc, its length and any doc boost.=0AI don't have access to the cod=
e right now but that is how I remember it working. There may be an option t=
o turn term frequency off too.   =0A=0A=0A=0AOn 27 Aug 2009, at 14:25, Berk=
es Adam <adam.berkes@intland.com> wrote:=0A=0AAfter searching for term "des=
y" which has lot of variants in our index a rewritten (sub)query will look =
like this:=0A=0A(text:dey^0.22828968 text:des^0.22828968 text:dest^1.155718=
4 text:desk^1.1557184 text:desi^1.1557184 text:desf^1.1557184 text:desc^1.1=
557184 text:deny^1.1557184 text:defy^1.1557184 text:desy^8.218443)=0A=0Abut=
 what I would like to achive to have all exact matches (even if rankings "v=
alidly" send it to the end of matches) on top (or highest possible) while l=
et variants to follow them according to their relevancy.=0A=0AMaybe I under=
stand wrongly but the edit distance is not a factor in that query type: ind=
ex is search for terms with edit distance within a certain limit, eliminate=
 IDF (with the factors above) and then create a coordinationless boolean qu=
ery. I might play around (post modify) scoring for exact match subterm but =
I'm not sure that is a working solution.=0A=0ABest regards,=0AAdam=0ADespit=
e making IDF a constant the edit distance should remain a factor in the ran=
kings so I would have thought this would give you what you need.=0A=0ACan y=
ou supply a more detailed example? Either print the rewritten query or use =
the explain function=0A=0ACheers=0AMark=0A=0AOn 27 Aug 2009, at 13:22, Berk=
es Adam wrote:=0A=0AHi,=0A=0AIn our java project we uses a (slightly modife=
d) version of FuzzyLikeThis query which=0A=0A"For each source term the fuzz=
y variants are held in a BooleanQuery with no coord factor (because=0Awe ar=
e not looking for matches on multiple variants in any one doc). Additionall=
y, a specialized=0ATermQuery is used for variants and does not use that var=
iant term's IDF because this would favour rarer=0Aterms eg misspellings. In=
stead, all variants use the same IDF ranking (the one for the source query=
=0Aterm) and this is factored into the variant's boost. If the source query=
 term does not exist in the=0Aindex the average IDF of the variants is used=
."=0A=0AIn most cases it performs well but if there is short query term wit=
h (as usual) big number of variants the exact matches will be stay spreaded=
 among the others which is not so useful: it should be "sorted" like (or fo=
rcibly set more relevant) exact matches and variant matches according to re=
levancy.=0AIs there any simple solution or already implemented contrib quer=
y class for this problem?=0A=0ABest regards,=0AAdam Berkes,=0AIntland Softw=
are=0A=0A------------------------------------------------------------------=
---=0ATo unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org=0AFor =
additional commands, e-mail: java-dev-help@lucene.apache.org=0A=0A=0A=0A---=
------------------------------------------------------------------=0ATo uns=
ubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org=0AFor additional c=
ommands, e-mail: java-dev-help@lucene.apache.org=0A=0A=0A=0A---------------=
------------------------------------------------------=0ATo unsubscribe, e-=
mail: java-dev-unsubscribe@lucene.apache.org=0AFor additional commands, e-m=
ail: java-dev-help@lucene.apache.org=0A=0A=0A=0A=0A      

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org