hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Dahiya (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-788) Streaming should use a subclass of TextInputFormat for reading text inputs.
Date Tue, 16 Jan 2007 11:41:27 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465134
] 

Sanjay Dahiya commented on HADOOP-788:
--------------------------------------

patch is attached for review. 

> Streaming should use a subclass of TextInputFormat for reading text inputs.
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-788
>                 URL: https://issues.apache.org/jira/browse/HADOOP-788
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: contrib/streaming
>            Reporter: Owen O'Malley
>         Assigned To: Sanjay Dahiya
>         Attachments: Hadoop-788.patch
>
>
> Currently streaming uses a lot of custom code for processing text inputs. 
> I propose:
>  1. Move class LineRecordReader  out of TextInputFormat.
>  2. Make class StreamLineRecordReader extend LineRecordReader.
>  3. StreamLineRecordReader uses LineRecordReader.next to read the lines and splits them
on tab to generate a Text/Text key/value pair.
> This will remove a lot of code from streaming and give it automatic support for the compression
codecs that the "base" part of Hadoop enjoys. In particular, if the native zlib code is used,
it will remove the 2gb limit on compressed files.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message