hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Fair scheduler.
Date Mon, 15 Oct 2012 22:18:14 GMT
Hi Patai,

Reply inline.

On Tue, Oct 16, 2012 at 2:57 AM, Patai Sangbutsarakum
<silvianhadoop@gmail.com> wrote:
> Thanks for input,
> I am reading the document; i forget to mention that i am on cdh3u4.

That version should have the support for all of this.

>> If you point your poolname property to mapred.job.queue.name, then you
>> can leverage the Per-Queue ACLs
> Is that mean if i plan to 3 pools of fair scheduler, i have to
> configure 3 queues of capacity scheduler. in order to have each pool
> can leverage Per-Queue ACL of each queue.?

Queues are not hard-tied into CapacityScheduler. You can have generic
queues in MR. And FairScheduler can bind its Pool concept into the
Queue configuration.

All you need to do is the following:

1. Map FairScheduler pool name to reuse queue names itself:

mapred.fairscheduler.poolnameproperty set to 'mapred.job.queue.name'

2. Define your required queues:

mapred.job.queues set to "default,foo,bar" for example, for 3 queues:
default, foo and bar.

3. Define Submit ACLs for each Queue:

mapred.queue.default.acl-submit-job set to "patai,foobar users,adm"
(usernames groupnames)

mapred.queue.foo.acl-submit-job set to "spam eggs"

Likewise for remaining queues, as you need it…

4. Enable ACLs and restart JT.

mapred.acls.enabled set to "true"

5. Users then use the right API to set queue names before submitting
jobs, or use -Dmapred.job.queue.name=value via CLI (if using Tool):

6. Done.

Let us know if this works!

Harsh J

View raw message