hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jothi Padmanabhan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3478) The algorithm to decide map re-execution on fetch failures can be improved
Date Tue, 03 Jun 2008 04:54:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601832#action_12601832
] 

Jothi Padmanabhan commented on HADOOP-3478:
-------------------------------------------

bq. Given that it is quite hard to sort by MapId for a given location

Actually, we do not need to sort the MapIds for a given location. We just need to ensure that
the order of map fetches is enforced -- All reducers fetch the maps in the same order as any
other. We could do this without sorting. 

bq.If there are a lot of unfinished mappers, then we should not do aggressive mapper re-executions.

Yes. This was proposed as a solution in HADOOP-3327. The JobTracker should use the number
of unfinished mappers as one of the criteria for deciding whether maps should be re-executed
aggressively or not.

bq. If reducers have a lot of un-fetched map outputs, they can wait for a longer period time
before re-fetch the map outputs that failed to fetcher previously.

The problem with this approach is that, if the map is faulty, it takes a long time to detect
and then re-execute it. If the detection is done earlier, the map will likely finish re-execution
by the time the reducer fetches other map outputs and the over all time is minimized.

> The algorithm to decide map re-execution on fetch failures can be improved
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-3478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3478
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Jothi Padmanabhan
>
> The algorithm to decide map re-execution on fetch failures can be improved.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message