mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: vector encoding of text documents
Date Thu, 10 Jan 2013 18:03:29 GMT
It has to do with a few things.

First, most classifiers can learn as good or better weights than TF-IDF
weighting, but k-means really needs the extra help.

Second, there is an issue of when the code was developed.  Most of the
clustering code was developed before the feature hashing stuff was
available.  Thus, the mismatch in techniques.  Clearly IDF could have been
used with feature hashing, although reversibility would suffer.

On Thu, Jan 10, 2013 at 9:21 AM, Koert Kuipers <koert@tresata.com> wrote:

> i noticed that the mahout encoding of text documents in a k-means example
> uses TF-IDF and then converts the documents into vectors (so one vector per
> document) where every word gets mapped to an int for the vector index and
> the words TFIDF score becomes the vector value for that index. it also
> creates a dictionary to be able to get back from the index to the word. did
> i get that right?
>
> why this approach as opposed to the feature hashing that i saw in other
> places in mahout?
>
> thanks! koert
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message