hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1184) mapred.reduce.slowstart.completed.maps is too high by default
Date Thu, 05 Nov 2009 04:21:32 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773768#action_12773768
] 

Matei Zaharia commented on MAPREDUCE-1184:
------------------------------------------

This is a good idea. Ideally though, we might actually want slow start to depend on the amount
of map output data and the rate at which data can be copied. If you have a job with only a
few MB of map output per reducer, setting slow start as high as 95% isn't going to impact
your response time too much. On the other hand, if you have a job where the maps "explode"
the output and you know that the bulk of your time will be spent in the shuffle phase, you
might want to set it lower.

> mapred.reduce.slowstart.completed.maps is too high by default
> -------------------------------------------------------------
>
>                 Key: MAPREDUCE-1184
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1184
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Allen Wittenauer
>
> By default, this value is set to 5%.  I believe for most real world situations the code
isn't efficient enough to be set this low.  This should be higher, probably around the 50%
mark, especially given the predominance of non-FIFO schedulers.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message