hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-2639) Reducers stuck in shuffle
Date Tue, 29 Jan 2008 02:07:34 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-2639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Arun C Murthy updated HADOOP-2639:

    Attachment: HADOOP-2639_1_20080128.patch

Ok, this is Amar's original patch after incorporating comments from Devaraj, poring with a
lens through findNewTask, obtainNew{Map|Reduce}Task and finally completedTask to ensure it
works; along with some helpful comments.

I've also reproduced the original problem (-ve values for running{Map|reduce}Tasks) via some
complicated shenanigans and also checked that this patch fixes them. 

Appreciate feedback while I'm running/monitoring some large-scale benchmarks on with this

> Reducers stuck in shuffle
> -------------------------
>                 Key: HADOOP-2639
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2639
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Amareshwari Sri Ramadasu
>            Assignee: Amar Kamat
>            Priority: Blocker
>             Fix For: 0.16.0
>         Attachments: HADOOP-2639.patch, HADOOP-2639_1_20080128.patch
> I started sort benchmark on 500 nodes. It has 40000 maps and 900 reducers.
> There are 11 reducers stuck in shuffle with 33% progress. I could see a node down which
ran 80 maps on it. And all these reducers are trying to fetch map output from that node. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message