hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3229) Map OutputCollector does not report progress on writes
Date Fri, 11 Apr 2008 20:08:05 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588098#action_12588098
] 

Chris Douglas commented on HADOOP-3229:
---------------------------------------

The flag is set by TrackedRecordReader for each record given to the map. Do we need to call
it for each record emitted? It's also set after each call to reduce in the combiner (not each
record; my mistake). I thought the problem was that the spill doesn't report progress after
the sort, without a combiner ("on writing"). As demonstrated in HADOOP-2284, the overhead
of setting this flag- as you assert- is slight, but not free.

If we wanted to set this flag after each record written, then we might as well add a SequenceFile::createWriter
method that takes a Progressable object and a FSDataOutputStream, and cover both cases.

> Map OutputCollector does not report progress on writes
> ------------------------------------------------------
>
>                 Key: HADOOP-3229
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3229
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>         Environment: all
>            Reporter: Alejandro Abdelnur
>             Fix For: 0.17.0
>
>         Attachments: 3229-0.patch, HADOOP-3229.patch
>
>
> It seem that the collector implementation used during the map phase does not report progress
on writing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message