Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of cambazz@gmail.com designates
 209.85.198.226 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:references;
        b=O4j6Iz5BYuxI2DF+b/5yPALeTJzRvmWVMY+HiuGvBa1wIJ5UzCWwBa0CKU74zZQRaC
         VEx2CC9GFMzF84fN7I7dSwi0+XDX/gXQagDwdJahUOKLQmFlGpfzGmXDJ+DY5+YsZIUu
         lk7PSJSBA7qUWx2iogCAeilatyRYCk2lhn0FM=
Message-ID: <1bcb7c7f0809040654u577fd444g2f3dea32c4f1adcd@mail.gmail.com>
Date: Thu, 4 Sep 2008 16:54:14 +0300
From: "Cam Bazz" <cambazz@gmail.com>
To: java-user@lucene.apache.org
Subject: Re: string similarity measures
In-Reply-To: <B23B5A00-EA22-42EC-BEF8-7F102474776F@gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_50646_9892424.1220536454996"
References: <1bcb7c7f0809040538s681fb81aud4e3e2aa19435caf@mail.gmail.com>
	 <B23B5A00-EA22-42EC-BEF8-7F102474776F@gmail.com>

------=_Part_50646_9892424.1220536454996
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

yes, I already have a system for users reporting words. they fall on an
operator screen and if operator approves, or if 3 other people marked it as
curse, then it is filtered.
in the other thread you wrote:

>I would create 1-5 ngram sized shingles and measure the distance using
Tanimoto coefficient. That would probably work out just fine. ?>You might
want to add more weight the greater the size of the shingle.
>
>There are shingle filters in lucene/java/contrib/analyzers and there is a
Tanimoto distance in lucene/mahout/.

would that apply to my case? tanimoto coefficient over shingles?

Best,


On Thu, Sep 4, 2008 at 4:12 PM, Karl Wettin <karl.wettin@gmail.com> wrote:

>
> 4 sep 2008 kl. 14.38 skrev Cam Bazz:
>
>
>  Hello,
>> This came up before but - if we were to make a swear word filter, string
>> edit distances are no good. for example words like `shot` is confused with
>> `shit`. there is also problem with words like hitchcock. appearently i
>> need
>> something like soundex or double metaphone. the thing is - these are
>> language specific, and i am not operating in english.
>>
>> I need a fuzzy like curse word filter for turkish, simply.
>>
>
> You probably need to make a large list of words. I would try to learn from
> the users that do swear, perhaps even trust my users to report each other. I
> would probably also look at storing in what context the word is used,
> perhaps by adding the surrounding words (ngrams, shingles, markov chains).
> Compare "go to hell" and "when hell frezes over". The first is rather
> derogatory while the second doen't have to be bad at all.
>
> I'm thinking Hidden Markov Models and Neural Networks.
>
>
>          karl
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

------=_Part_50646_9892424.1220536454996--