hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Johan Oskarsson (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-4666) Launch reduces only after a few maps have run in the Fair Scheduler
Date Mon, 24 Nov 2008 11:30:44 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-4666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Johan Oskarsson updated HADOOP-4666:

       Resolution: Fixed
    Fix Version/s: 0.20.0
         Assignee: Matei Zaharia
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I just committed this. Thanks Matei!

> Launch reduces only after a few maps have run in the Fair Scheduler
> -------------------------------------------------------------------
>                 Key: HADOOP-4666
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4666
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: contrib/fair-share
>            Reporter: Matei Zaharia
>            Assignee: Matei Zaharia
>             Fix For: 0.20.0
>         Attachments: fairscheduler-reduce-launch-condition.patch
> It makes no sense to schedule reduces for a job before its maps have started running.
As an initial fix, we will wait until a certain percent have run (likely 10%). In the future
it would be good to choose the time to wait based on amount of map output data as well - launching
reducers that will mostly be idle is not helpful. Average amount of map output bytes per mapper
is easy to compute using counters in JobInProgress.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message