hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zheng Shao (JIRA)" <j...@apache.org>
Subject [jira] Created: (MAPREDUCE-1649) Compressed files with TextInputFormat does not work with CombineFileInputFormat
Date Tue, 30 Mar 2010 18:36:27 GMT
Compressed files with TextInputFormat does not work with CombineFileInputFormat
-------------------------------------------------------------------------------

                 Key: MAPREDUCE-1649
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1649
             Project: Hadoop Map/Reduce
          Issue Type: Bug
    Affects Versions: 0.20.2
            Reporter: Zheng Shao
            Assignee: Zheng Shao


{{CombineFileInputFormat}} creates splits based on blocks, regardless whether the underlying
{{FileInputFormat}} is splittable or not..

This means that we can have 2 or more splits for a compressed text file with {{TextInputFormat}}.
For each of these splits, {{TextInputFormat.getRecordReader}} will return a {{RecordReader}}
for the whole compressed file, thus causing duplicate input data.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message