lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Willnauer <>
Subject Re: Lucene TermVector
Date Tue, 22 Feb 2011 08:02:20 GMT

On Mon, Feb 21, 2011 at 8:56 PM, Ajay Anandan <> wrote:
> Hi
> I am trying to implement an Expectation Maximization algorithm for document
> clustering.  I am planning to use Lucene Term Vectors for finding similarity
> between 2 documents.  There are 2 kinds of EM algos using naive Bayes: the
> multivariate model and the multinomial model.  In simple terms, the
> multinomial model uses the frequencies of different words in the documents
> which the multivariate model just uses the info of whether a word is present
> or not in the document(a boolean vector).
> I know that the term vectors in Lucene store the terms present in the
> current document along with their frequencies.  This is exactly what is
> needed for the multinomial model.
> But the multivariate model requires the following:
> A vector which stores the presence or absence of a particular term.  Thus
> all the terms in all the documents must be handled by this vector.
> As an example:
> doc1 : field CONTENT has the following terms : this is the world of
> pleasure.
> doc2 : field CONTENT has the following terms : this amazing world is full of
> sarcastic people.
> now the vector that I need should be
> < this is the world of pleasure amazing full sarcastic people > ( it
> contains all the words in all the documents )
> for doc1 the value of this vector is <1 1 1 1 1 1 0 0 0 0>
> for doc2 the vakue of this vector is <1 1 0 1 0 0 1 1 1 1>
> Is there any way to generate such a boolean vector in Lucene?

Lucene stores as you found correctly a sparse representation of
TermVectors along with their frequencies, positions and / or offsets
if needed. The terms which occur in other document but are not in a
particular one are omitted (they might even not be present when the
vector is stored). Yet, what lucene offers you is one big vector with
all unique terms for each field, its term dictionary. you can simply
build your dense vector as needed at runtime. You can simply traverse
the TermEnum for that field and the TermVector at the same time. for
each term in the TermEnum not present in you vector you insert a 0.

Maybe there is a better way to do it or already existing
infrastructure for this I don't know about just a possibility.

> --
> Ajay Anandan.
> MSc,computing Science, University of Alberta.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message