Mailing-List: contact mahout-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mahout-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of milesosb@gmail.com designates
 209.85.210.192 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=references:message-id:from:to:in-reply-to:content-type
         :content-transfer-encoding:x-mailer:mime-version:subject:date:cc;
        b=SrkpMhtWmmqt8/0d5Mo/D4lTiU0I8VT94D30+0AQ3fsbpt1wt3PV0v0eHq7FQQ3eUC
         GdzjG6Xen0SxtGpyXji0woAgwVdFei9VvuDcUsrxKlfkifU3VHXj8pNUbvo+TBicngL3
         /vDV8FEE/BstACY3lnhyfo2z12igaB2iy1DWw=
References: <5E00122C-1C77-4623-922E-867502CD09AA@apache.org>
 <89776790-590D-4080-A76B-3482D57556AE@transpac.com>
 <c7d45fc70911131106w7b5db93au6148a05787ce17b5@mail.gmail.com>
Message-Id: <9797DDF9-805E-4EA9-BE40-89F638F4D3CF@gmail.com>
From: Miles <milesosb@gmail.com>
To: "mahout-user@lucene.apache.org" <mahout-user@lucene.apache.org>
In-Reply-To: <c7d45fc70911131106w7b5db93au6148a05787ce17b5@mail.gmail.com>
Content-Type: text/plain;
	charset=us-ascii;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (iPod Mail 7C145)
Subject: Re: http://bixolabs.com/datasets/public-terabyte-dataset-project/
Date: Fri, 13 Nov 2009 19:48:32 +0000
Cc: "mahout-user@lucene.apache.org" <mahout-user@lucene.apache.org>

A very simple way to remove boilerplate (and this is trivial using Map  
Reduce) is to just remove all duplicate sentences. This does assume  
you can extract sentences, do sentence boundary detection etc.

Miled
Sent from your Ipod


On 13 Nov 2009, at 19:06, Ted Dunning <ted.dunning@gmail.com> wrote:

> This looks like a very nice approach for getting rid of the goo.  I  
> often
> advocate using words/phrases/ngrams that are highly predicted by the  
> domain
> name as an alternative for removing boilerplate.  That has the  
> advantage
> that it doesn't require training text.  In the case of wiki-pedia,  
> this is
> not so useful because everything is in the same domain.  The domain
> predictor trick will only work if the feature you are using for the  
> input is
> not very content based.  Thus, this can fail for small domain- 
> focused sites
> or if you use a content laden URL for the task.
>
>
>
> On Fri, Nov 13, 2009 at 10:36 AM, Ken Krugler
> <kkrugler_lists@transpac.com>wrote:
>
>> Hi all,
>>
>> Another issue came up, about cleaning the text.
>>
>> One interested user suggested using nCleaner (see
>> http://www.lrec-conf.org/proceedings/lrec2008/pdf/885_paper.pdf) as  
>> a way
>> of tossing boilerplate text that skews text frequency data.
>>
>> Any thoughts on this?
>>
>> Thanks,
>>
>> -- Ken
>>
>>
>> On Nov 3, 2009, at 5:43am, Grant Ingersoll wrote:
>>
>> Might be of interest to all you Mahouts out there...
>>> http://bixolabs.com/datasets/public-terabyte-dataset-project/
>>>
>>> Would be cool to get this converted over to our vector format so  
>>> that we
>>> can cluster, etc.
>>>
>>
>> --------------------------------------------
>> Ken Krugler
>> +1 530-210-6378
>> http://bixolabs.com
>> e l a s t i c   w e b   m i n i n g
>>
>>
>>
>>
>>
>
>
> -- 
> Ted Dunning, CTO
> DeepDyve