hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4666) Launch reduces only after a few maps have run in the Fair Scheduler
Date Fri, 21 Nov 2008 23:17:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649844#action_12649844

dhruba borthakur commented on HADOOP-4666:

+1. Code looks good.

> Launch reduces only after a few maps have run in the Fair Scheduler
> -------------------------------------------------------------------
>                 Key: HADOOP-4666
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4666
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: contrib/fair-share
>            Reporter: Matei Zaharia
>         Attachments: fairscheduler-reduce-launch-condition.patch
> It makes no sense to schedule reduces for a job before its maps have started running.
As an initial fix, we will wait until a certain percent have run (likely 10%). In the future
it would be good to choose the time to wait based on amount of map output data as well - launching
reducers that will mostly be idle is not helpful. Average amount of map output bytes per mapper
is easy to compute using counters in JobInProgress.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message