Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of manjula53@gmail.com designates
 209.85.212.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=xVWsx3dL22GGAJlM0jdpgCBpDwwtDJTyc5i/DwRNjUKDHP8kIyLpbfYSycms73elmP
         3alpsTAhhCbckFMDU/Eu8QT6wQ9MIwiMAi33auK+VWl/UoD0IxNclD6WtUGC9Z1kxvzA
         Hpwk9XXVY82W4YTKazYWacbk5EP72PXsvQ6vk=
MIME-Version: 1.0
In-Reply-To: <025701cb1f39$f55f46d0$e01dd470$@thetaphi.de>
References: <AANLkTin3uDNGvlEzSr_53i3HXROyCeb8WhZBo3QfpVnK@mail.gmail.com>
	<AANLkTimD4pUFeTcyNcEsNV8c5f7b0wex7N49-8qRllFZ@mail.gmail.com>
	<AANLkTin3TAaxEL26oq_ejIW9lSAK2CE0HIMn1z3p6dld@mail.gmail.com>
	<025701cb1f39$f55f46d0$e01dd470$@thetaphi.de>
Date: Fri, 9 Jul 2010 16:00:45 +0530
Message-ID: <AANLkTimpmCw83AfM9seXpkZ-VOUd1jjLx0Yw4mXvEDCL@mail.gmail.com>
Subject: Re: Why not normalization?
From: manjula wijewickrema <manjula53@gmail.com>
To: java-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=e0cb4e887eb3548e5d048af1e521

--e0cb4e887eb3548e5d048af1e521
Content-Type: text/plain; charset=ISO-8859-1

Thanx

On Fri, Jul 9, 2010 at 1:10 PM, Uwe Schindler <uwe@thetaphi.de> wrote:

> > Thanks for your valuble comments. Yes I observed tha, once the number of
> > terms of the goes up, fieldNorm value goes down correspondingly. I think,
> > therefore there won't be any default due to the variation of total number
> of
> > terms in the document. Am I right?
>
> With the current scoring model advanced statistics are not available. There
> are currently some approaches to add BM25 support to Lucene, for what the
> index format needs to be enhanced to contain more statistics (number of
> terms per document, avg number of terms per document,...).
>
> > On Thu, Jul 8, 2010 at 9:34 AM, Rebecca Watson <bec.watson@gmail.com>
> > wrote:
> >
> > > hi,
> > >
> > > > 1) Although Lucene uses tf to calculate scoring it seems to me that
> > > > term frequency has not been normalized. Even if I index several
> > > > documents, it does not normalize tf value. Therefore, since the
> > > > total number of words in index documents are varied, can't there be
> > > > a fault in Lucene's
> > > scoring?
> > >
> > > tf = term frequency i.e. the number of times the term appears in the
> > > document, while idf is inverse document frequency - is a measure of
> > > how rare a term is, i.e. related to how many documents the term
> > > appears in.
> > >
> > > if term1 occurs more frequently in a document i.e. tf is higher, you
> > > want to weight the document higher when you search for term1
> > >
> > > but if term1 is a very frequent term, ie. in lots of documents, then
> > > its probably not as important to an overall search (where we have
> > > term1, term2 etc) so you want to downweight it (idf comes in)
> > >
> > > then the normalisations like length normalisation (allow for 'fair'
> > > scoring across varied field length) come in too.
> > >
> > > the tf-idf scoring formula used by lucene is a  scoring method that's
> > > been around a long long time... there are competing scoring metrics
> > > but that's an IR thing and not an argument you want to start on the
> > > lucene lists! :)
> > >
> > > these are IR ('information retrieval') concepts and you might want to
> > > start by going to through the tf-idf scoring / some explanations for
> > > this kind of scoring.
> > >
> > > http://en.wikipedia.org/wiki/Tf%E2%80%93idf
> > > http://wiki.apache.org/lucene-java/InformationRetrieval
> > >
> > >
> > > > 2) What is the formula to calculate this fieldNorm value?
> > >
> > > in terms of how lucene implements its tf-idf scoring - you can see
> here:
> > > http://lucene.apache.org/java/3_0_2/scoring.html
> > >
> > > also, the lucene in action book is a really good book if you are
> > > starting out with lucene (and will save you a lot of grief with
> > > understanding lucene / setting up your application!), it covers all
> > > the basics and then moves on to more advanced stuff and has lots of
> > > code examples too:
> > > http://www.manning.com/hatcher2/
> > >
> > > hope that helps,
> > >
> > > bec :)
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

--e0cb4e887eb3548e5d048af1e521--