hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-3940) Reduce often attempts in memory merge with no work
Date Wed, 13 Aug 2008 21:44:44 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Chris Douglas updated HADOOP-3940:

    Attachment: 3940-1.patch

Added some comments to waitForDataToMerge, particularly explaining why numRequiredMapOutputs
== 0 is a special case, per Arun's suggestion.

Test failures are due to HADOOP-3946.

> Reduce often attempts in memory merge with no work
> --------------------------------------------------
>                 Key: HADOOP-3940
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3940
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Chris Douglas
>            Priority: Minor
>         Attachments: 3940-0.patch, 3940-1.patch
> ReduceTask.ReduceCopier.ShuffleRamManager initializes numRequiredMapOutputs to 0, so
one of the predicates in ShuffleRamManager::waitForDataToMerge, {{numPendingRequests <
numRequiredMapOutputs}}, is false until the first map output is fetched and false again after
the last map output is fetched. The InMemFSMergeThread thread will loop busily in this state.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message