Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding;
  b=2VjaV5Jvi+s4R5BZx03B6E+0zL75/E7KS3l8EPKK4TLeVzyOhTne3jb9rF4LBijXyEfaYf0v3tmsza2GhXiSrxGCZLJEyzOUlzUhWvEx3SCOp0sZPlG1UY1RRMkP03c2w34KIKU5PV+38Rv9+Lvl3GVZ6Ptk7oUTS4pwTXvUeMQ=;
Message-ID: <713587.43972.qm@web113318.mail.gq1.yahoo.com>
Date: Mon, 3 May 2010 17:41:53 -0700 (PDT)
From: Ivan Provalov <iprovalo@yahoo.com>
Subject: Re: Relevancy Practices
To: java-user@lucene.apache.org
In-Reply-To: <E6C119F7-C966-4280-94DD-3575BB3CE3F4@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Grant,=0A=0AWe are currently working on a relevancy improvement project.  W=
e took the IBM's paper from 2007 TREC and followed the approaches they desc=
ribed to improve Lucene's relevance.  It also gave us some idea of Lucene=
=E2=80=99s out-of-the-box precision performance (MAP).  In addition to it w=
e used some of the best practices described in TREC's book (Voorhees 2005, =
MIT).  We also looked into the probability scoring model (BM25). =0A=0AWe s=
tarted by comparing =E2=80=9Cvanilla=E2=80=9D Lucene to our Lucene-based pr=
oduct=E2=80=99s performance.  We obtained the collections and the judgments=
 from the past TREC which were close to the genre of the content we store. =
 We then proceeded to study how different tunings affected the scores.  We =
used Lucene's benchmarking module to run against the TREC data.  Even thoug=
h there were a few old TREC document/topic format related issues along the =
way, this benchmarking tool was all together great in helping find the MAP =
and measure where we were at.  =0A=0AThen we applied the Sweet Spot similar=
ity, Pivot Point document length normalization (Lnb/Ltc), and BM25 scoring =
algorithms.  After applying these different scoring mechanism changes and o=
ther techniques (different stemmers, query expansion), we saw some improvem=
ents.  We then compared this to our current production system and started t=
uning it as well.  =0A=0AOur second goal here was to include the relevance =
measurement into the continuous integration tests running nightly.  The tho=
ught here is that if one of the system=E2=80=99s changes inadvertently affe=
cted the scoring, we would find out right away.  This second phase also hel=
ped us discover hidden bugs in our production system. =0A=0AIn addition to =
the English-based analyzers, we also studied Chinese analyzers and compared=
 the results with the English collection runs.  We used TREC data for that.=
=0A=0ASome observations:=0A1.=09Even though the Vector Space model with Boo=
lean query (OR) gives good MAP scores, in some products the large number of=
 returned results makes the product less usable.  So, defaulting to AND ope=
rator may be a better option as was mentioned in this user group post earli=
er.=0A2.=09This TREC-based evaluation is just of many tools to use.  For ex=
ample, user feed-back is still the most important evaluation one can do.=0A=
3.=09We will continue studying how different scoring mechanisms affect rele=
vance quality before making a decision whether to switch from the default V=
SM.  Some of our concerns are over-tuning and performance testing.=0A4.=09L=
ucene user community has been very helpful.  Robert Muir, Joaquin Iglesias,=
 and others helped with applying the scoring algorithms and providing great=
 suggestions. =0A5.=09Some of the tools we use constantly - Lucene=E2=80=99=
s query Explanation and Luke.=0A=0AThanks,=0A=0AIvan Provalov=0A=0A=0A=0A=
=0A--- On Thu, 4/29/10, Grant Ingersoll <gsingers@apache.org> wrote:=0A=0A>=
 From: Grant Ingersoll <gsingers@apache.org>=0A> Subject: Relevancy Practic=
es=0A> To: java-user@lucene.apache.org=0A> Date: Thursday, April 29, 2010, =
10:14 AM=0A> I'm putting on a talk at Lucene=0A> Eurocon (http://lucene-eur=
ocon.org/sessions-track1-day2.html#1)=0A> on "Practical Relevance" and I'm =
curious as to what people=0A> put in practice for testing and improving rel=
evance.=C2=A0 I=0A> have my own inclinations, but I don't want to muddy the=
=0A> water just yet.=C2=A0 So, if you have a few moments, I'd=0A> love to h=
ear responses to the following questions.=0A> =0A> What worked?=C2=A0 =0A> =
What didn't work?=C2=A0 =0A> What didn't you understand about it?=C2=A0 =0A=
> What tools did you use?=C2=A0 =0A> What tools did you wish you had either=
 for debugging=0A> relevance or "fixing" it?=0A> How much time did you spen=
d on it?=0A> How did you avoid over/under tuning?=0A> What stage of develop=
ment/testing/production did you decide=0A> to do relevance tuning?=C2=A0 Wa=
s that timing planned or=0A> not?=0A> =0A> =0A> Thanks,=0A> Grant=0A> =0A=
=0A=0A      

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org