hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1463) Reducer should start faster for smaller jobs
Date Sat, 06 Feb 2010 10:26:28 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830502#action_12830502
] 

Amar Kamat commented on MAPREDUCE-1463:
---------------------------------------

Scott, 
- How do you define small jobs. Shouldnt it be based on total number of tasks instead of considering
maps and reduces individually? 
- Why do we need special case for small jobs? If its for fairness then this piece of code
rightly belongs to contrib/fairscheduler, no?
- If not for fairness then what is the problem with the current framework w.r.t small jobs?
- Can be fixed by simple (configuration-like) tweaking?
- If not then whats the right fix. 

Wouldn't the reducers be scheduled faster if 'mapreduce.job.reduce.slowstart.completedmaps'
is set to 0? If not then can we change the slowstart feature to get it right?

> Reducer should start faster for smaller jobs
> --------------------------------------------
>
>                 Key: MAPREDUCE-1463
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1463
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/fair-share
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>         Attachments: MAPREDUCE-1463-v1.patch, MAPREDUCE-1463-v2.patch
>
>
> Our users often complain about the slowness of smaller ad-hoc jobs.
> The overhead to wait for the reducers to start in this case is significant.
> It will be good if we can start the reducer sooner in this case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message