hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-3940) Reduce often attempts in memory merge with no work
Date Tue, 12 Aug 2008 21:17:45 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Chris Douglas updated HADOOP-3940:

    Attachment: 3940-0.patch

Simplest fix, waiting on numRequiredMapOutputs == 0.

Initializing numRequiredMapOutputs to Integer.MAX_VALUE would require a different check at
the end; this should handle both.

> Reduce often attempts in memory merge with no work
> --------------------------------------------------
>                 Key: HADOOP-3940
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3940
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Chris Douglas
>            Priority: Minor
>         Attachments: 3940-0.patch
> ReduceTask.ReduceCopier.ShuffleRamManager initializes numRequiredMapOutputs to 0, so
one of the predicates in ShuffleRamManager::waitForDataToMerge, {{numPendingRequests <
numRequiredMapOutputs}}, is false until the first map output is fetched and false again after
the last map output is fetched. The InMemFSMergeThread thread will loop busily in this state.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message