hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Johan Oskarsson (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4666) Launch reduces only after a few maps have run in the Fair Scheduler
Date Mon, 24 Nov 2008 12:40:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650190#action_12650190

Johan Oskarsson commented on HADOOP-4666:

The tests failures were not related to the patch. I've created this issue for one of them
that I've seen time out in a few other patch runs. HADOOP-4716

> Launch reduces only after a few maps have run in the Fair Scheduler
> -------------------------------------------------------------------
>                 Key: HADOOP-4666
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4666
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: contrib/fair-share
>            Reporter: Matei Zaharia
>            Assignee: Matei Zaharia
>             Fix For: 0.20.0
>         Attachments: fairscheduler-reduce-launch-condition.patch
> It makes no sense to schedule reduces for a job before its maps have started running.
As an initial fix, we will wait until a certain percent have run (likely 10%). In the future
it would be good to choose the time to wait based on amount of map output data as well - launching
reducers that will mostly be idle is not helpful. Average amount of map output bytes per mapper
is easy to compute using counters in JobInProgress.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message