Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Message-Id: <DF1BB6F3-8BC9-414F-A415-2F75FFFCEE86@apache.org>
From: Grant Ingersoll <gsingers@apache.org>
To: java-user@lucene.apache.org
In-Reply-To: 
 <OFA433B746.CDF7AA2D-ON652575D0.002D6853-652575D0.002D7C07@lntinfotech.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: Using Lucene for Moderate Similarity Check..
Date: Tue, 9 Jun 2009 10:34:01 -0400
References: 
 <OFA433B746.CDF7AA2D-ON652575D0.002D6853-652575D0.002D7C07@lntinfotech.com>

Hi Ravi,

Lucene can enable this, but you will have some work to do on top of  
it.  If you search the archives for record linkage (http://www.lucidimagination.com/search/?q=record+linkage 
) you will find a fair amount of discussion on this.  Also, in  
somewhat shameless marketing mode, my co-author, Tom Morton, is just  
putting the finishing touches on a chapter in our book called Taming  
Text (http://www.manning.com/ingersoll) which discusses some of the  
techniques involved in making this stuff happen.  That chapter should  
be released in the next few weeks.

You likely can get a basic system working pretty quickly with what is  
in Lucene, but then the next level is often more difficult.  You often  
end up with a rules system that can become brittle with this  
approach.  An alternative is to apply some type of machine learning  
approach.  You could also look at this as a clustering problem, which  
Mahout (or other clustering tools) could be helpful in solving.

Finally, just know there will be a human in the loop with any  
approach.  The goal is to minimize the number of matches that a person  
has to check.

Hope this helps,
Grant

On Jun 9, 2009, at 4:16 AM, RaviK Thakur wrote:

>
> Hello All,
>      I want to check the feasibility of using Lucene for similarity  
> check
> between the two flat csv files. The actual requirement is like this:  
> We
> have two files each containing the information of customers like their
> name, address, pin code etc. Some customers may be in common in both  
> the
> files. We want to find the customer that are common in these files.  
> But the
> match should be on attribute basis. If the name of the customer  
> matches in
> one file to the name of the customer in another file, then match the
> address, if it matches then match pin code and so on. But the main
> consideration is that this matching is not exact. If the name of  
> customer
> matches say 80% then it may be termed as match. For example, if  
> ABDUL is
> matched with ABDULLAH, it should be termed as a match. In this  
> fashion each
> record of one file will be matched with each record of another file.  
> The
> output of this procedure will be another file containing the matched
> record.
>
> Can anyone please suggest the applicability of lucene for this  
> requirement.
> May in the form of Pros n Cons.
>
> Thanks in advance:-)
> Ravi
>
>
> ______________________________________________________________________
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

--------------------------
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)  
using Solr/Lucene:
http://www.lucidimagination.com/search


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org