hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4666) Launch reduces only after a few maps have run in the Fair Scheduler
Date Sun, 23 Nov 2008 09:19:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650000#action_12650000

Hadoop QA commented on HADOOP-4666:

-1 overall.  Here are the results of testing the latest attachment 
  against trunk revision 719787.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 5 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3637/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3637/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3637/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3637/console

This message is automatically generated.

> Launch reduces only after a few maps have run in the Fair Scheduler
> -------------------------------------------------------------------
>                 Key: HADOOP-4666
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4666
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: contrib/fair-share
>            Reporter: Matei Zaharia
>         Attachments: fairscheduler-reduce-launch-condition.patch
> It makes no sense to schedule reduces for a job before its maps have started running.
As an initial fix, we will wait until a certain percent have run (likely 10%). In the future
it would be good to choose the time to wait based on amount of map output data as well - launching
reducers that will mostly be idle is not helpful. Average amount of map output bytes per mapper
is easy to compute using counters in JobInProgress.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message