hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-372) should allow to specify different inputformat classes for different input dirs for Map/Reduce jobs
Date Mon, 21 Jul 2008 09:47:34 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615185#action_12615185
] 

Alejandro Abdelnur commented on HADOOP-372:
-------------------------------------------

I find this feature useful but I'm not sure about how it has been implemented (apologies for
the late comment, just become aware of it when seeing the Jira Resolved email).

*{{FileInputFormat}} API changes:*

All the methods added to the {{FileInputFormat}} are not related to {{InputFormat}} functionality.

In my opinion they should be in a separate class (something like {{MultipleInputs}} )

*Package for all the patch classes:*

All these classes should be in {{org.apache.hadoop.mapred.lib}} thus keeping the core {{mapred}}
with exactly that, the core, this is a useful extension but still is an extension.

*{{JobConf}} for the different mappers:*

The current code forces all mappers to have the same {{JobConf}}, this means that mappers
have to be written with knowledge of the other mappers to avoid collision in the configuration
keys.

How about passing along a {{JobConf}} instance when adding an input path with its own {{Mapper}},
something like HADOOP-3702 is doing when adding a mapper to the chain?


> should allow to specify different inputformat classes for different input dirs for Map/Reduce
jobs
> --------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-372
>                 URL: https://issues.apache.org/jira/browse/HADOOP-372
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.4.0
>         Environment: all
>            Reporter: Runping Qi
>            Assignee: Chris Smith
>             Fix For: 0.19.0
>
>         Attachments: hadoop-372.patch, hadoop-372.patch, hadoop-372.patch, hadoop-372.patch,
hadoop-372.patch
>
>
> Right now, the user can specify multiple input directories for a map reduce job. 
> However, the files under all the directories are assumed to be in the same format, 
> with the same key/value classes. This proves to be  a serious limit in many situations.

> Here is an example. Suppose I have three simple tables: 
> one has URLs and their rank values (page ranks), 
> another has URLs and their classification values, 
> and the third one has the URL meta data such as crawl status, last crawl time, etc. 
> Suppose now I need a job to generate a list of URLs to be crawled next. 
> The decision depends on the info in all the three tables.
> Right now, there is no easy way to accomplish this.
> However, this job can be done if the framework allows to specify different inputformats
for different input dirs.
> Suppose my three tables are in the following directory respectively: rankTable, classificationTable.
and metaDataTable. 
> If we extend JobConf class with the following method (as Owen suggested to me):
>     addInputPath(aPath, anInputFormatClass, anInputKeyClass, anInputValueClass)
> Then I can specify my job as follows:
>     addInputPath(rankTable, SequenceFileInputFormat.class, UTF8.class, DoubleWritable.class)
>     addInputPath(classificationTable, TextInputFormat.class, UTF8,class, UTF8.class)
>     addInputPath(metaDataTable, SequenceFileInputFormat.class, UTF8.class, MyRecord.class)
> If an input directory is added through the current API, it will have the same meaning
as it is now. 
> Thus this extension will not affect any applications that do not need this new feature.
> It is relatively easy for the M/R framework to create an appropriate record reader for
a map task based on the above information.
> And that is the only change needed for supporting this extension.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message