hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-2639) Reducers stuck in shuffle
Date Sat, 26 Jan 2008 01:55:35 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-2639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Arun C Murthy updated HADOOP-2639:

    Status: Open  (was: Patch Available)

HADOOP-2247 introduced code which relied on a buggy counter, so that was buggy indeed.

I'm not super sure, but looks to me with this patch that running{Maps/Reduces}Tasks are incremented
more than once if we have speculative tasks, so it bothers me. 

Like Owen commented, there is a reason why this code is so - maybe the fix is to figure out
why {{numRunningReduces}} is getting negative? That is why I think we can fix this differently...

> Reducers stuck in shuffle
> -------------------------
>                 Key: HADOOP-2639
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2639
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Amareshwari Sri Ramadasu
>            Assignee: Amar Kamat
>            Priority: Blocker
>             Fix For: 0.16.0
>         Attachments: HADOOP-2639.patch
> I started sort benchmark on 500 nodes. It has 40000 maps and 900 reducers.
> There are 11 reducers stuck in shuffle with 33% progress. I could see a node down which
ran 80 maps on it. And all these reducers are trying to fetch map output from that node. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message