hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hong Tang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1184) mapred.reduce.slowstart.completed.maps is too high by default
Date Thu, 05 Nov 2009 06:09:32 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773798#action_12773798

Hong Tang commented on MAPREDUCE-1184:

I think Allen's point is that the default 5% may be too low from the utilization perspective.

My point (which may be shared with Matei) is that this really could be adaptively tuned by
the MR framework (thus eliminating the need of a configuration knob).  Finally, back to my
comment on turn around time, i think users should specify high level optimization objectives
such as whether they care more about response time or throughput, and MR framework should
adjust related parameters automatically. Granted, this is probably beyond the scope of this

> mapred.reduce.slowstart.completed.maps is too high by default
> -------------------------------------------------------------
>                 Key: MAPREDUCE-1184
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1184
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Allen Wittenauer
> By default, this value is set to 5%.  I believe for most real world situations the code
isn't efficient enough to be set this low.  This should be higher, probably around the 50%
mark, especially given the predominance of non-FIFO schedulers.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message