Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <308566893.1218178844480.JavaMail.jira@brutus>
Date: Fri, 8 Aug 2008 00:00:44 -0700 (PDT)
From: "Vuk Ercegovac (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Created: (HADOOP-3926) Multiple, generic InputFormats for
 MapReduce
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Multiple, generic InputFormats for MapReduce
--------------------------------------------

                 Key: HADOOP-3926
                 URL: https://issues.apache.org/jira/browse/HADOOP-3926
             Project: Hadoop Core
          Issue Type: Improvement
          Components: mapred
            Reporter: Vuk Ercegovac
            Priority: Minor


The feature that allows an InputFormat per path to be specified for a MapRe=
duce job should be generalized
(see http://issues.apache.org/jira/browse/HADOOP-372) to support InputForma=
ts other than FileInputFormat
(e.g., HBase table). This is needed when joining or co-grouping multiple in=
puts. Even for the case of multiple FileInputFormats, it seems that if a su=
b-class sets and configures itself from the JobConf, the inputs will need t=
o insure that they do not have name clashes. In general, the child InputFor=
mats should not be aware of each other.

We've implemented this for Jaql but would like to remove dependencies on ot=
her libs (json) and see how it can be integrated with the HADOOP-372 change=
s. It works similar to HADOOP-372. A UnionInputFormat consists of multiple =
child InputFormats. The UnionInputFormat records an array of <InputFormat, =
name-value pairs for JobConf> in the JobConf. For creating splits, it colle=
cts child splits (similar to DelegatingInputFormat) and wraps each child's =
split with its index into the array (similar to TaggedInputSplit). The Unio=
nInputFormat, given a split, can then dig out the corresponding InputFormat=
 given its index, instantiate it, and return its RecordReader. Each child I=
nputFormat depends on setting up an empty JobConf prior to its instantiatio=
n. An alternative is to use a string version of an InputFormat's setup JobC=
onf. The analog to DelegatingMapper simply exposes the child split's index =
to drive per input logic (in our case, its a script rather than a Map class=
). As with HADOOP-372, these are lib-level changes, not core.

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.