hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1184) mapred.reduce.slowstart.completed.maps is too high by default
Date Thu, 05 Nov 2009 05:44:32 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773791#action_12773791

Matei Zaharia commented on MAPREDUCE-1184:

Yeah, actually the 5% setting can be a source of latency for small jobs in my experience,
because the maps will finish at roughly the same time, and you then need to wait a few seconds
for a reducer to start up and to get the map completion events from the JobTracker. For these
jobs, it might make sense to look at the rate at which maps are reporting progress and launch
the reducers when it looks like the map will finish in the next 5 seconds. There are many
other things that could be done to decrease the latency for small jobs however.

> mapred.reduce.slowstart.completed.maps is too high by default
> -------------------------------------------------------------
>                 Key: MAPREDUCE-1184
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1184
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Allen Wittenauer
> By default, this value is set to 5%.  I believe for most real world situations the code
isn't efficient enough to be set this low.  This should be higher, probably around the 50%
mark, especially given the predominance of non-FIFO schedulers.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message