Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of jpforny@gmail.com designates
 74.125.83.52 as permitted sender)
MIME-Version: 1.0
From: =?ISO-8859-1?Q?Jo=E3o_Paulo_Forny?= <jpforny@gmail.com>
Date: Fri, 28 Feb 2014 14:37:01 -0300
Message-ID: 
 <CAMttJ_-baJy2xviD=cPDz6isO8TagjG8uuMXTxhoMxqHiw-0uw@mail.gmail.com>
Subject: Reduce side join of similar records
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=047d7bdc8c8e0c27e704f37ae506

--047d7bdc8c8e0c27e704f37ae506
Content-Type: text/plain; charset=ISO-8859-1

I'm implementing a join between two datasets A and B by a String key, which
is the name attribute. I need to match similar names in this join.

My first thought, given that I was implementing secondary sort to get the
values extracted from database A before the values from database B, was to
create a grouping comparator class and instead of using the compareTo
method to group values by the natural key, use a string similarity
algorithm, but it has not worked as expected, since that names that match
in my algorithm wasn't mapped as the same key. See my code below.

public class StringSimilarityGroupingComparator extends WritableComparator {

protected StringSimilarityGroupingComparator() {
    super(JoinKeyTagPairWritable.class, true);
}

public int compare(WritableComparable w1, WritableComparable w2) {
    JoinKeyTagPairWritable k1 = (JoinKeyTagPairWritable) w1;
    JoinKeyTagPairWritable k2 = (JoinKeyTagPairWritable) w2;
    StringSimilarityMatcher nameMatcher = new StringSimilarityMatcher(
            StringSimilarityMatcher.NAME_MATCH);

    return nameMatcher.match(k1.getJoinKey(), k2.getJoinKey()) ? 0 : k1
            .getJoinKey().compareTo(k2.getJoinKey());
}

This approach makes total sense to me. Where was I mistaken? Isn't this the
purpose of overriding the grouping comparator class?

--047d7bdc8c8e0c27e704f37ae506
Content-Type: text/html; charset=ISO-8859-1

<div dir="ltr">I&#39;m implementing a join between two datasets A and B by a String key, 
which is the name attribute. I need to match similar names in this join.

<p>My first thought, given that I was implementing secondary sort to get
 the values extracted from database A before the values from database B,
 was to create a grouping comparator class and instead of using the 
compareTo method to group values by the natural key, use a string 
similarity algorithm, but it has not worked as expected, since that names that match in my algorithm wasn&#39;t mapped as the same key. See my code 
below.</p>

<pre><code>public class StringSimilarityGroupingComparator extends WritableComparator {

protected StringSimilarityGroupingComparator() {
    super(JoinKeyTagPairWritable.class, true);
}

public int compare(WritableComparable w1, WritableComparable w2) {
    JoinKeyTagPairWritable k1 = (JoinKeyTagPairWritable) w1;
    JoinKeyTagPairWritable k2 = (JoinKeyTagPairWritable) w2;
    StringSimilarityMatcher nameMatcher = new StringSimilarityMatcher(
            StringSimilarityMatcher.NAME_MATCH);

    return nameMatcher.match(k1.getJoinKey(), k2.getJoinKey()) ? 0 : k1
            .getJoinKey().compareTo(k2.getJoinKey());
}
</code></pre>

<p>This approach makes total sense to me. Where was I mistaken? Isn&#39;t this the purpose of overriding the grouping comparator class?</p></div>

--047d7bdc8c8e0c27e704f37ae506--