hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3229) Map OutputCollector does not report progress on writes
Date Fri, 11 Apr 2008 20:58:05 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588113#action_12588113

Chris Douglas commented on HADOOP-3229:

bq. If a slow-running mapper or reducer only outputs one tiny record every minute or so...

Shouldn't the progress update from TrackedRecordReader take care of that? I suppose there
could be a map that reads a single record and emits several, small records over several minutes...
OK, sold. For most cases, that sounds like a dead/dying node that probably _should_ get killed,
but even if that were true this would be the wrong way to address it.

bq. that's the contract we've advertised in the past, that either consuming or emitting an
entry counted as task progress

Setting the flag on collect from the map, as above, I agree. We're missing a case by only
following collect(), though: should we signal progress while we're spilling to disk, without
a combiner? Passing the Reporter to the fs::create would be address tasks with/without a combiner,

> Map OutputCollector does not report progress on writes
> ------------------------------------------------------
>                 Key: HADOOP-3229
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3229
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>         Environment: all
>            Reporter: Alejandro Abdelnur
>             Fix For: 0.17.0
>         Attachments: 3229-0.patch, HADOOP-3229.patch
> It seem that the collector implementation used during the map phase does not report progress
on writing.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message