lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Halácsy Péter <halacsy.pe...@axelero.com>
Subject RE: Normalization of Documents
Date Thu, 11 Apr 2002 14:43:16 GMT


> -----Original Message-----
> From: Peter Carlson [mailto:carlson@bookandhammer.com]
> Sent: Thursday, April 11, 2002 4:35 PM
> To: Lucene Users List
> Subject: Re: Normalization of Documents
> 
> 
> Hi,
> 
> These types of questions/discussions should be on the users 
> list, not dev
> list, please.
> 
OK

> 
> Just for the record, the Lucene scoring is not as simple as just a %.
> From the FAQ.
> 
> For the record, Lucene's scoring algorithm is, roughly:
> 
>   score_d = sum_t( tf_q * idf_t / norm_q * tf_d * idf_t / norm_d_t)
What I would like:

  score_d = sum_t( tf_q * idf_t / norm_q * tf_d * idf_t / norm_d_t) * p_value_d

where:
p_value_d : predefined value of document calculated at indexing time (0 < p_value_d <=
1)

in the API:
option 1:
writer = new IndexWriter(..)
writer.addDocument(doc, 0.45);

option 2 (I think better)

Document d = new Document();
d.setValue(0.45);
d.addField(..);
writer.addDocument();

peter



>  
> where:
>   score_d   : score for document d
>   sum_t     : sum for all terms t
>   tf_q      : the square root of the frequency of t in the query
>   tf_d      : the square root of the frequency of t in d
>   idf_t     : log(numDocs/docFreq_t+1) + 1.0
>   numDocs   : number of documents in index
>   docFreq_t : number of documents containing t
>   norm_q    : sqrt(sum_t((tf_q*idf_t)^2))
>   norm_d_t  : square root of number of tokens in d in the 
> same field as t
> 
> (I hope that's right!)
> 
> [Doug later added...]
> 
> Make that:
>   
>   score_d = sum_t(tf_q * idf_t / norm_q * tf_d * idf_t / 
> norm_d_t * boost_t)
> * coord_q_d
> 
> where
> 
>   boost_t    : the user-specified boost for term t
>   coord_q_d  : number of terms in both query and document / 
> number of terms
> in query
> 
> The coordination factor gives an AND-like boost to documents 
> that contain,
> e.g., all three terms in a three word query over those that 
> contain just two
> of the words.
> 
> 
> 
> Although this may still not be what you want. You should be 
> able to replace
> the scoring mechanism with your own. The problem you might run into is
> getting the document data (such as date) will slow down your 
> search speed
> dramatically.
> 
> Do you know about any solutions (Academic or free) that 
> provide this concept
> extraction. I've heard of a group in the UK who worked on 
> something like
> this.
> 
> --Peter
> 
> 
> 
> On 4/11/02 6:51 AM, "Halácsy Péter" <halacsy.peter@axelero.com> wrote:
> 
> > Extracting concept is not too easy thing and I don't think 
> you can implement a
> > language/context/document type independent solution. 
> Filtering only important
> > terms of a text (and not index all text as in modern full 
> text indexing
> > system) is one of the most important area of IR. A lot of 
> project worked on
> > this topic but nowadays it's not too important because we 
> can index every
> > terms if we want (cheaper and faster disk, lot of CPU).
> > 
> > I think in lucene the the term's % of the document
> > (NUMBER_OF_WORDS_IN_THE_DOCUMENT / 
> NUMBER_OF_QUERY_TERM_ACCURENCE )is
> > overweighted in some case. I would like to tune it if I could.
> > 
> > Document scoring could provide solution for me and I think 
> for Melissa as
> > well. I think it's a very important feature of a modern IR 
> system. For example
> > Melissa would use it to score the documents based on link 
> popularity (or
> > impact factor/citation frequency). In my project I should 
> score documents on
> > their length and their age (more recent document is more 
> valuable and very old
> > documents are as valuable as very new in my archive).
> > 
> > peter
> > 
> >> -----Original Message-----
> >> From: Peter Carlson [mailto:carlson@bookandhammer.com]
> >> Sent: Wednesday, April 10, 2002 5:17 PM
> >> To: Lucene Developers List
> >> Subject: Re: Normalization of Documents
> >> 
> >> 
> >> I have noticed the same issue.
> >> 
> >> From what I understand, this is both the way it should work
> >> and a problem.
> >> Shorter documents which have a given term, should be more
> >> relevant because
> >> more of the document is about that term (i.e the term takes a
> >> greater % of
> >> the document). However, when there are documents of
> >> completely different
> >> sizes (i.e. 20 words vs. 2000 words) this assumption doesn't
> >> hold up very
> >> well.
> >> 
> >> One solution I've heard is to extract the concepts of the
> >> documents, then
> >> search on those. The concepts are still difficult to extract
> >> if the document
> >> is too short, but it may provide a way to standardize
> >> documents. I have been
> >> lazily looking for an open source, academic concept
> >> extractor, but I haven't
> >> found one. There are companies like Semio and
> >> ActiveNavigation which provide
> >> this service for an expense fee.
> >> 
> >> Let me know if you find anything or have other ideas.
> >> 
> >> --Peter
> >> 
> >> 
> >> On 4/9/02 10:07 PM, "Melissa Mifsud" 
> <melissamifsud@yahoo.com> wrote:
> >> 
> >>> Hi,
> >>> 
> >>> Documents which are shorter in length always seem to score
> >> higher in Lucene. I
> >>> was under the impression that the nornalization factors in
> >> the scoring
> >>> function used by Lucene would improve this, however, after
> >> a couple of
> >>> experiments, the short documents still always score the highest.
> >>> 
> >>> Does anyone have any ideas as to how it is possible to make
> >> lengthier
> >>> documents score higher?
> >>> 
> >>> Also, I would like a way to boost documents according to
> >> the amount of
> >>> in-links this document has.
> >>> 
> >>> Has anyone implemented a type of Document.setBoost() method?
> >>> 
> >>> I found a thread in the lucene-dev mailinglist where Doug
> >> Cutting mentions
> >>> that this would be a great feature to add to Lucene. No one
> >> followed his
> >>> email.
> >>> 
> >>> Melissa.
> >>> 
> >> 
> >> 
> >> --
> >> To unsubscribe, e-mail:
> >> <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> >> For additional commands, e-mail:
> >> <mailto:lucene-dev-help@jakarta.apache.org>
> >> 
> >> 
> > 
> > --
> > To unsubscribe, e-mail:   
> <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> > For additional commands, e-mail: 
> <mailto:lucene-dev-help@jakarta.apache.org>
> > 
> > 
> 
> 
> --
> To unsubscribe, e-mail:   
> <mailto:lucene-user-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail: 
> <mailto:lucene-user-help@jakarta.apache.org>
> 
> 

--
To unsubscribe, e-mail:   <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>


Mime
View raw message