hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sameer Paranjpye (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-343) In case of dead task tracker, the copy mapouts try copying all mapoutputs from this tasktracker
Date Wed, 04 Oct 2006 16:59:23 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-343?page=comments#action_12439897 ] 
Sameer Paranjpye commented on HADOOP-343:

This patch addresses the concern raised. If a map output transfer from a particular tasktracker
fails, other output locations from the tasktracker that are present in the cache are removed.
This addresses the problem of repeated attempts and backoffs from a lost tasktracker, which
is particularly bad towards the end of a shuffle. Copies can, of course, fail for other reasons,
in these cases also output locations are removed. The cost of this removal is fairly low.
This is because the number of output locations cached for a specific tasktracker is usually
small (3-4), and removing these (multiple times even) results in a handful of extra polls
of the jobtracker.

> In case of dead task tracker, the copy mapouts try copying all mapoutputs from this tasktracker
> -----------------------------------------------------------------------------------------------
>                 Key: HADOOP-343
>                 URL: http://issues.apache.org/jira/browse/HADOOP-343
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.6.2
>            Reporter: Mahadev konar
>         Assigned To: Sameer Paranjpye
>             Fix For: 0.7.0
>         Attachments: bugfix.patch, cache-purge.txt
> In case of a dead task tracker, the reduces which do not have the updated map out locations
try copygin files from this node and since there are failures on copying, this leads to backoff
and slowing down of the copy pahse.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message