hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5670) CombineFileRecordReader should report progress when moving to the next file
Date Thu, 05 Dec 2013 17:42:37 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13840323#comment-13840323
] 

Jason Lowe commented on MAPREDUCE-5670:
---------------------------------------

CombineFileRecordReader.initNextRecordReader should report progress as it moves to the next
file to indicate that the task is making progress even though no records are being read. 
Some input formats already report progress when their record reader is created, but adding
a progress indicator here will catch the cases where the underlying record reader does not
report progress upon construction.

> CombineFileRecordReader should report progress when moving to the next file
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5670
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5670
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 0.23.9, 2.2.0
>            Reporter: Jason Lowe
>            Priority: Minor
>
> If a combine split consists of many "empty" files (i.e.: no record found by the underlying
record reader) then theoretically a task can timeout due to lack of reported progress.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message