Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
Message-ID: <5959599.1173917349532.JavaMail.jira@brutus>
Date: Wed, 14 Mar 2007 17:09:09 -0700 (PDT)
From: "Runping Qi (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Created: (HADOOP-1120) Contribute some code helping
 implement map/reduce apps for joining data from multiple sources
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Contribute some code helping implement map/reduce apps for joining data from multiple sources
---------------------------------------------------------------------------------------------

                 Key: HADOOP-1120
                 URL: https://issues.apache.org/jira/browse/HADOOP-1120
             Project: Hadoop
          Issue Type: New Feature
          Components: contrib/streaming
            Reporter: Runping Qi


With the current Hadoop, it is a bit hard for the user to implement data joining apps. 
HADOOP-475/485 attempt to provide some support for data joining jobs, but it seems to be had to implement.

This Jira rather calls for a application level support. 
The idea is to provide a generic map/reduce classes implementing data join jobs, 
and allows the user to extend those classes to add their special logic. 

In particular, the user needs to define a mapper class 
that extends DataJoinMapperBase class  to implement methods for the
following functionalities:

1. Compute the source tag of input values 
2. Compute the map output value object 
3. Compute the map output key object
 
The source tag will be used by the reducer to determine from which source
(which table in SQL terminology) a value comes. Computing the map output
value object amounts to performing projecting/filtering work in a SQL
statement (through the select/where clauses). Computing the map output key
amounts to choosing the join key. This class provides the appropriate plugin
points for the user defined subclasses to implement the appropriate logic.

The the user needs to define a reducer class 
that extends DataJoinReduceBase class  to implement the following:

    protected abstract TaggedMapOutput combine(Object[] tags, Object[] values);
 
The above method is expected to produce one output value from an array of
records of different sources. The user code can also perform filtering here.
It can return null if it decides to the records do not meet certain conditions.

That is pretty much the user need to do in order to create a map/reduce job to join data 
from different sources.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.