Mailing-List: contact mahout-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mahout-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of jake.mannix@gmail.com
 designates 209.85.160.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=sMTdoyguCFt8E8WEzv3nsIcrwMobMMQ9CIxbX7c+sy2ohvITzXjT37CE+2l/4cjLmA
         6OlxoAcObNfcmn45BSNuH+XU+kIMMCZ/fLXSdvT0B8ojI3JVgVHrUEkNpA4ZkV6/WrVJ
         tzoyKIBCGAIPQhtcSn69qOq2S+f2MXpQMUmAE=
MIME-Version: 1.0
In-Reply-To: <4b124c311002160959p7a70cfddv861ee05b1d313f7c@mail.gmail.com>
References: <8f8e14c41002160828y44c12defxf4bd9f95b6ae087b@mail.gmail.com>
	 <4b124c311002160954kfff36adn7404bb6b224ea79f@mail.gmail.com>
	 <4b124c311002160958p273653d7l78fc615c7add11ee@mail.gmail.com>
	 <4b124c311002160959p7a70cfddv861ee05b1d313f7c@mail.gmail.com>
Date: Tue, 16 Feb 2010 10:03:07 -0800
Message-ID: <4b124c311002161003y6abfbe0cga84799430188c685@mail.gmail.com>
Subject: Re: n-gram over-representation?
From: Jake Mannix <jake.mannix@gmail.com>
To: mahout-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=0016e64ce5c2ccf0c4047fbb8b52

--0016e64ce5c2ccf0c4047fbb8b52
Content-Type: text/plain; charset=ISO-8859-1

Drew,

  Did you pick your whitelist using the LLR score?  What is the kind of
over-representation you're trying to prune out?  DF will certainly help you
remove "too common" bigrams, but that's not what you're looking for, is it?

  -jake

On Feb 16, 2010 8:29 AM, "Drew Farris" <drew.farris@gmail.com> wrote:

I have a collection of about 800k bigrams from a corpus of 3.7m
documents that I'm in the process of working with. I'm looking to
determine an appropriate subset of these to use both as features for
both an ML and an IR application. Specifically I'm considering
white-listing a subset of these to use as features when building a
classifier and separately as terms when building an index and doing
query parsing. As a part of the earlier collocation discussion Ted
mentioned that tests for over-representation could be used to identify
dubious members of such a set.

Does anyone have any pointers to discussions of how such a test could
be implemented?

Thanks,

Drew

--0016e64ce5c2ccf0c4047fbb8b52--