mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jake Mannix <>
Subject Re: n-gram over-representation?
Date Tue, 16 Feb 2010 18:03:07 GMT

  Did you pick your whitelist using the LLR score?  What is the kind of
over-representation you're trying to prune out?  DF will certainly help you
remove "too common" bigrams, but that's not what you're looking for, is it?


On Feb 16, 2010 8:29 AM, "Drew Farris" <> wrote:

I have a collection of about 800k bigrams from a corpus of 3.7m
documents that I'm in the process of working with. I'm looking to
determine an appropriate subset of these to use both as features for
both an ML and an IR application. Specifically I'm considering
white-listing a subset of these to use as features when building a
classifier and separately as terms when building an index and doing
query parsing. As a part of the earlier collocation discussion Ted
mentioned that tests for over-representation could be used to identify
dubious members of such a set.

Does anyone have any pointers to discussions of how such a test could
be implemented?



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message