hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-4666) Launch reduces only after a few maps have run in the Fair Scheduler
Date Fri, 21 Nov 2008 22:59:44 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-4666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Matei Zaharia updated HADOOP-4666:

    Attachment: fairscheduler-reduce-launch-condition.patch

Here is a patch for this issue that waits for at least 5% of maps to be launched, as will
be happening in HADOOP-3136. (Patch also includes some minor formatting fixes.)

> Launch reduces only after a few maps have run in the Fair Scheduler
> -------------------------------------------------------------------
>                 Key: HADOOP-4666
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4666
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: contrib/fair-share
>            Reporter: Matei Zaharia
>         Attachments: fairscheduler-reduce-launch-condition.patch
> It makes no sense to schedule reduces for a job before its maps have started running.
As an initial fix, we will wait until a certain percent have run (likely 10%). In the future
it would be good to choose the time to wait based on amount of map output data as well - launching
reducers that will mostly be idle is not helpful. Average amount of map output bytes per mapper
is easy to compute using counters in JobInProgress.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message