Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of kasunp@opensource.lk designates
 209.85.210.46 as permitted sender)
MIME-Version: 1.0
From: Kasun Perera <kasunp@opensource.lk>
Date: Fri, 20 Apr 2012 13:50:19 +0530
Message-ID: 
 <CALGBXaNbmhBkFO5OgNxy551KRxJmfN0BaNjG=W2sPhcHMoLyaQ@mail.gmail.com>
Subject: Weighted cosine similarity calculation using Lucene
To: java-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=047d7b10cfefc2d57f04be17f635

--047d7b10cfefc2d57f04be17f635
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

I have documents that are marked up with Taxonomy and Ontology terms
separately.
When I calculate the document similarity, I want to give higher weights to
those Taxonomy terms and Ontology terms.


When I index the document, I have defined the Document content, Taxonomy
and Ontology terms as Fields for each document like this in my program.


*Field ontologyTerm=3D new Field("fiboterms", fiboTermList[curDocNo],
Field.Store.YES, Field.Index.ANALYZED, Field.TermVector.YES);*

*Field taxonomyTerm =3D new Field("taxoterms", taxoTermList[curDocNo],
Field.Store.YES, Field.Index.ANALYZED, Field.TermVector.YES);*

*Field document =3D new Field(docNames[curDocNo], strRdElt,
Field.TermVector.YES);*


I=92m using Lucene index .TermFreqVector functions to calculate TFIDF value=
s
and, then calculate cosine similarity between two documents using TFIDF
values.


For give weights to Ontology and Taxonomy terms when calculating the cosine
similarity, what I can do is, programmatically multiply the Taxonomy
and Ontology
term frequencies with defined weight factor before calculating the TFIDF
scores. Will this give higher weight to Taxonomy and Ontology terms in
document similarity calculation?


Are there Lucene functions that can be used to give higher weights to the
certain fields when calculating TFIDF values using TermFreqVector? can I
just use the setboost() function for this purpose, then how?

--=20
Regards

Kasun Perera

--047d7b10cfefc2d57f04be17f635--