hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Runping Qi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3130) Shuffling takes too long to get the last map output.
Date Tue, 01 Apr 2008 16:14:24 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12584198#action_12584198

Runping Qi commented on HADOOP-3130:

Speaking of failing reducer because of failing to fetch map output, we got to do some careful
analysis here.
At least, we have to differentiate between the case of failing to fetch one map output numerous
times and the case of failing to
fetch a lot of different map outputs. In the first case, it is better to re-execute the map.

In the second case, maybe it makes sense to consider to fail the reducer.

Also, we should differentiate between the early stage of shuffling (where the reducer may
have thousands of map outputs to fetch)
and the late stage where only a few map outputs are left for fetching. In the early stage,
it does not matter to fail to connect to a 
few mappers, since the reducer has plenty to do.  In the late stage, failing the reducer is
much costly than re-execute the maps.

> Shuffling takes too long to get the last map output.
> ----------------------------------------------------
>                 Key: HADOOP-3130
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3130
>             Project: Hadoop Core
>          Issue Type: Bug
>            Reporter: Runping Qi
>         Attachments: HADOOP-3130.patch, shuffling.log
> I noticed that towards the end of shufflling, the map output fetcher of the reducer backs
off too aggressively.
> I attach a fraction of one reduce log of my job.
> Noticed that the last map output was not fetched in 2 minutes.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message