hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3940) Reduce often attempts in memory merge with no work
Date Tue, 12 Aug 2008 22:07:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622004#action_12622004

Chris Douglas commented on HADOOP-3940:

Alternatively, I think this could equivalently wait on {{numPendingRequests <= numRequiredMapOutputs}},
relying on close to signal the last merge

> Reduce often attempts in memory merge with no work
> --------------------------------------------------
>                 Key: HADOOP-3940
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3940
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Chris Douglas
>            Priority: Minor
>         Attachments: 3940-0.patch
> ReduceTask.ReduceCopier.ShuffleRamManager initializes numRequiredMapOutputs to 0, so
one of the predicates in ShuffleRamManager::waitForDataToMerge, {{numPendingRequests <
numRequiredMapOutputs}}, is false until the first map output is fetched and false again after
the last map output is fetched. The InMemFSMergeThread thread will loop busily in this state.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message