lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ajay Anandan <ajayda...@gmail.com>
Subject Lucene TermVector
Date Mon, 21 Feb 2011 19:56:30 GMT
Hi
I am trying to implement an Expectation Maximization algorithm for document
clustering.  I am planning to use Lucene Term Vectors for finding similarity
between 2 documents.  There are 2 kinds of EM algos using naive Bayes: the
multivariate model and the multinomial model.  In simple terms, the
multinomial model uses the frequencies of different words in the documents
which the multivariate model just uses the info of whether a word is present
or not in the document(a boolean vector).

I know that the term vectors in Lucene store the terms present in the
current document along with their frequencies.  This is exactly what is
needed for the multinomial model.

But the multivariate model requires the following:
A vector which stores the presence or absence of a particular term.  Thus
all the terms in all the documents must be handled by this vector.

As an example:

doc1 : field CONTENT has the following terms : this is the world of
pleasure.

doc2 : field CONTENT has the following terms : this amazing world is full of
sarcastic people.

now the vector that I need should be

< this is the world of pleasure amazing full sarcastic people > ( it
contains all the words in all the documents )

for doc1 the value of this vector is <1 1 1 1 1 1 0 0 0 0>

for doc2 the vakue of this vector is <1 1 0 1 0 0 1 1 1 1>

Is there any way to generate such a boolean vector in Lucene?

-- 

Ajay Anandan.
MSc,computing Science, University of Alberta.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message