hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo (Nicholas), SZE (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-706) Support for FIFO pools in the fair scheduler
Date Fri, 14 Aug 2009 22:07:14 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743468#action_12743468
] 

Tsz Wo (Nicholas), SZE commented on MAPREDUCE-706:
--------------------------------------------------

The design doc is very nice (especially, it was typeset by tex)!

Some suggestions for future works:
- In hadoop, fs usually refers to FileSystem.  "fsdesigndoc" sounds like FileSystem design
doc.  I think we should prevent overloading the term "fs".
- The tex file needs a license header.
- We do not have pdf files under ./src before.  "fsdesigndoc.pdf" is the first.  I think the
correct approach is to generate the pdf file by "ant docs".  However, it may not be easy to
do so.

> Support for FIFO pools in the fair scheduler
> --------------------------------------------
>
>                 Key: MAPREDUCE-706
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-706
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: contrib/fair-share
>            Reporter: Matei Zaharia
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: fsdesigndoc.pdf, fsdesigndoc.tex, mapreduce-706.patch, mapreduce-706.v1.patch,
mapreduce-706.v2.patch, mapreduce-706.v3.patch, mapreduce-706.v4.patch, mapreduce-706.v5.patch
>
>
> The fair scheduler should support making the internal scheduling algorithm for some pools
be FIFO instead of fair sharing in order to work better for batch workloads. FIFO pools will
behave exactly like the current default scheduler, sorting jobs by priority and then submission
time. Pools will have their scheduling algorithm set through the pools config file, and it
will be changeable at runtime.
> To support this feature, I'm also changing the internal logic of the fair scheduler to
no longer use deficits. Instead, for fair sharing, we will assign tasks to the job farthest
below its share as a ratio of its share. This is easier to combine with other scheduling algorithms
and leads to a more stable sharing situation, avoiding unfairness issues brought up in MAPREDUCE-543
and MAPREDUCE-544 that happen when some jobs have long tasks. The new preemption (MAPREDUCE-551)
will ensure that critical jobs can gain their fair share within a bounded amount of time.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message