hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1463) Reducer should start faster for smaller jobs
Date Tue, 16 Feb 2010 20:19:28 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834435#action_12834435

Todd Lipcon commented on MAPREDUCE-1463:

Stepping back a bit to think about the model. Correct me if you disagree:

Our end goal is that the reducers finish fetching reduce output as soon as possible after
the last mapper finishes, but that the reducers are started as late as possible, so they don't
occupy slots and hurt utilization.

So, let's assume that the mappers generate data at some rate M. The reducers can fetch data
at some maximum rate R. It's the ratio of M/R that determines when to start the reducers fetching.
For example, if the mappers generate data faster than the reducers can fetch it, it behooves
us to start the reduce fetch immediately when the job starts. If the reducers can fetch twice
as fast as the mappers can output, we want to start the reducers halfway through the map phase.

Since both kinds of tasks have some kind of startup cost, it's as if the average rate is slowed
down by a factor that's determined by the number of tasks. In the case of 200 mappers and
1 reducers, it's as if the map output speed has been lowered (since the fixed costs of the
map tasks slow down map completion), and thus we can afford to wait until later to start the
reducer. If you have 1 mapper and 1 reducer, even for the exact same job, the ratio swings
as if the map side output faster, and thus we want to start the reduce early.

This is of course a much simplified model, but I think it's worth discussing this on somewhat
abstract terms before we discuss the implementation details. One factor I'm ignoring above
is the limiting that the reducer does with respect to particular hosts - that is to say, 
the reducer fetch speed varies with the number of unique hosts, not just the number of mappers.

> Reducer should start faster for smaller jobs
> --------------------------------------------
>                 Key: MAPREDUCE-1463
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1463
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>         Attachments: MAPREDUCE-1463-v1.patch, MAPREDUCE-1463-v2.patch, MAPREDUCE-1463-v3.patch
> Our users often complain about the slowness of smaller ad-hoc jobs.
> The overhead to wait for the reducers to start in this case is significant.
> It will be good if we can start the reducer sooner in this case.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message