hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Runping Qi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3327) Shufflinge fetachers waited too long between map output fetch re-tries
Date Fri, 02 May 2008 07:28:58 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12593730#action_12593730
] 

Runping Qi commented on HADOOP-3327:
------------------------------------


How do you know when the fetch of a map output was scheduled first?

Why do you even need to confirm the time for first notification?
It is obvious that the re-try/backoff strategy is flawed.
Instead of following the schedule described above, the reducer should consider 
how many outstanding map outputs it still needs.
If not many map outputs need to be fetched, the reducer should not back off that long.
Also, the job tracker should decide whether to re-execute  a map based on how many fetch failure
AND how busy the system is.
If there are very few running mappers, then it should re-execute maps more aggressively.
 

> Shufflinge fetachers waited too long between map output fetch re-tries
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-3327
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3327
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Runping Qi
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message