hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From s d <sdano...@gmail.com>
Subject Capacity Scheduler questions
Date Sun, 06 Jan 2013 21:31:02 GMT
We are evaluating the Capacity Scheduler…

We would like to configure the equivalent of Fair Scheduler
userMaxJobsDefault = 1 (i.e. we would like to limit a user to a single job
in the cluster).

·         By default the Capacity Scheduler allows multiple jobs from a
single user to run concurrently.

·         From
appear to be limits for “the number of accepted/active jobs per user”.
However, the example capacity-scheduler.xml only has limits for active
tasks e.g. <queue>.maximum-initialized-active-tasks-per-user property.

·         Also the source CapacitySchedulerConf.java includes the following
code which suggests that the maximum jobs per user can be configured via
the init-accept-jobs-factor property. However, this is not clear from the
description of this property.

*  public int getInitToAcceptJobsFactor(String queue) {*

*    int initToAccepFactor =*

*      rmConf.getInt(toFullPropertyName(queue, "init-accept-jobs-factor"),*

*          defaultInitToAcceptJobsFactor);*

*    if(initToAccepFactor <= 0) {*

*      throw new IllegalArgumentException(*

*          "Invalid maximum jobs per user configuration " +

*    }*

*    return initToAccepFactor;*

*  }*

·         Also, other posts and sample xml files on the web refer to
property. However, I’ve tried setting this to 1 but it has no impact.

So… how can we configure the Capacity Scheduler to limit a user to a single
job in the cluster?



Also, I’m curious… a benefit of the Capacity Scheduler is that resource
limits can be specified in percentage terms, so if the cluster size changed
the CS configuration would not have to change. Therefore, why are some
properties specified in terms of tasks e.g.
which would need to be reconfigured if the cluster size changed?

View raw message