hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5659) Fair share schduler may support preemption only with a specific pool
Date Thu, 11 Jun 2009 23:15:07 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718665#action_12718665

Matei Zaharia commented on HADOOP-5659:

Hi Dhruba,

A question about this feature: How do you want to decide when to kill the wellcare job? Do
you want the tasks specified as a %, as a fixed number, or what?


> Fair share schduler may support preemption only with a specific pool
> --------------------------------------------------------------------
>                 Key: HADOOP-5659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5659
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: contrib/fair-share
>            Reporter: dhruba borthakur
> There are a set of jobs that helps to keep the cluster resources being used optimally.
For example, there are data sets that are made of a multiple files in a directory. These part-xxx
files could be concatenated to a relatively few files (to reduce memory  pressure on the namenode).
Also, there are files that could be compressed more efficiently (e.g. bzip2) to reduce save
on disk usage. These are kind of system-wellcare jobs that should run only if it does not
impact any other "real" user of the cluster. On an idle cluster, these wellcare jobs should
use all availale system resources. When a real user submits a job, the wellcare job(s) should
be pre-empted. If a scheduler can support pre-emption only for jobs in a specified pool, then
I can submit these well-care jobs to that special pool. Real user's jobs will never get pre-empted;but
the wellcare jobs can get pre-empted as soon as there is resource contention. If a task of
well-care jobs is pre-empted more than a configured max, the entire wellcare job will fail..
that this is the behaviour I want. The wellcare jobs would run in idle slots as long as all
user-submitted jobs have been satisfied, but would be preempted as soon as user jobs require
any of those slots.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message