hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5187) Provide an option to turn off priorities in jobs
Date Mon, 09 Feb 2009 07:31:02 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671753#action_12671753

Matei Zaharia commented on HADOOP-5187:

Right, this is what I was proposing. For fair sharing mode, the global ordering by deficit
works fine because it makes sure that each pool gets its own share. However, for FIFO, it
means that the scheduler as a whole runs jobs in FIFO order, regardless of what pool they're
in, so things like per-pool minimum shares don't work. I imagine it may be useful to support
having multiple pools each of which runs FIFO - this would emulate the capabilities of the
capacity scheduler for example. Does this make sense?

Doing this would currently require tricky logic if we wanted to integrate it with the deficit
model to allow FIFO and fair-share pools to coexist. However, if instead we remove the concept
of deficits and sort jobs by how far they are below their fair share (or how long they've
waited if they are below their min share), I think it should be possible to define a comparator
that takes into account both FIFO and fair-sharing pools. This is why I'm proposing to remove
the deficit model in HADOOP-4803. It seemed like a good idea initially, especially when the
fair scheduler had no preemption, but it does the wrong thing at times (as pointed out in
that JIRA) and it complicates the code, making it hard to add features such as this.

> Provide an option to turn off priorities in jobs
> ------------------------------------------------
>                 Key: HADOOP-5187
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5187
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: contrib/fair-share
>            Reporter: Hemanth Yamijala
>            Priority: Minor
> The fairshare scheduler can define pools mapping to queues (as defined in the capacity
scheduler - HADOOP-3445). When used in this manner, one can imagine queues set up to be used
by users who come from disparate teams or organizations (say a default queue). For such a
queue, it makes sense to ignore job priorities and consider the queue as strict FIFO, as it
is difficult to compare priorities of jobs from different users. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message