hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "madhukara phatak (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-8521) Port StreamInputFormat to new Map Reduce API
Date Fri, 22 Jun 2012 10:11:42 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-8521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

madhukara phatak updated HADOOP-8521:
-------------------------------------

    Attachment: HADOOP-8521-1.patch

I am a bit confused here. I see that you added in a new mapreduce StreamInputFormat, with
the corresponding StreamXmlRecordReader and StreamBaseRecordReader, but how does this enable
us to use the new MapReduce API? 
Can you update the documentation to provide some examples of how you can use these new classes
you have added?

>> New Map Reduce API requires InputFormat class to extend org.apache.hadoop.mapreduce.InputFormat
but StreamInputFormat is extending org.apache.hadoop.mapred.InputFormat.So when I try to set
it in Job as below
 job.setInputFormatClass(StreamInputFormat.class);
it gives compilation error. More info here http://search-hadoop.com/m/evL3S1deWQ72 .
So when I refer new API, i mean that porting the StreamInputFormat to new InputFormat class
, so that it can be used with new API code.
  
Also the test you have added in is not actually testing the new code at all. It is still testing
the old input format code. I can delete the new code entirely and the test still passes. It
looks like a great start, but I think there is some more wiring that needs to be done to make
this work.
>> updated the test case 

                
> Port StreamInputFormat to new Map Reduce API
> --------------------------------------------
>
>                 Key: HADOOP-8521
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8521
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: madhukara phatak
>            Assignee: madhukara phatak
>         Attachments: HADOOP-8521-1.patch, HADOOP-8521.patch
>
>
> As of now , hadoop streaming uses old Hadoop M/R API. This JIRA ports it to the new M/R
API.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message