Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 35BCE933C for ; Wed, 25 Jan 2012 14:39:40 +0000 (UTC) Received: (qmail 22442 invoked by uid 500); 25 Jan 2012 14:39:36 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 22355 invoked by uid 500); 25 Jan 2012 14:39:35 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 22347 invoked by uid 99); 25 Jan 2012 14:39:35 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Jan 2012 14:39:35 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of harsh@cloudera.com designates 209.85.160.48 as permitted sender) Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Jan 2012 14:39:30 +0000 Received: by pbcc11 with SMTP id c11so155466pbc.35 for ; Wed, 25 Jan 2012 06:39:10 -0800 (PST) Received: by 10.68.73.105 with SMTP id k9mr40514016pbv.121.1327502349179; Wed, 25 Jan 2012 06:39:09 -0800 (PST) MIME-Version: 1.0 Received: by 10.143.76.10 with HTTP; Wed, 25 Jan 2012 06:38:49 -0800 (PST) In-Reply-To: References:

From: Harsh J Date: Wed, 25 Jan 2012 20:08:49 +0530 Message-ID: Subject: Re: Understanding fair schedulers To: common-user@hadoop.apache.org X-Gm-Message-State: ALoCoQnAsRzWk8XWtzuk0e9kZUHK0J7mHPzHJXDrAjT63uMKjh+IPrNM033Xuwv8eKhfoDmJljhu Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable A solution would be to place your users into groups, and use group.name identifier to be the poolnameproperty. Would this work for you instead? On Wed, Jan 25, 2012 at 8:00 PM, praveenesh kumar wr= ote: > Also, with the above mentioned method, my problem is I am having one > pool/user (thats obviously not a good way of configuring schedulers) > How can I allocate multiple users to one pool in the xml properties, so > that I don't have to care giving any options inside my codes. > > Thanks, > Praveenesh > > On Wed, Jan 25, 2012 at 7:55 PM, praveenesh kumar w= rote: > >> I am looking for the solution where we can do it permanently without >> specify these things inside jobs. >> I want to keep these things hidden from the end-user. >> End-user would just write pig scripts and all the jobs submitted by the >> particular user will get submit to their respective pools automatically. >> >> What I am doing write now is something like this >> >> =A0 >> =A0 >> =A0 =A0 10 >> =A0 =A0 10 >> =A0 =A0 192 >> =A0 =A0 96 >> =A0 =A0 300 >> =A0 >> =A0 >> >> =A0 =A0 6 >> =A0 >> =A0 3 >> =A0 600 >> >> =A0 >> =A0 =A0 10 >> =A0 =A0 10 >> =A0 =A0 192 >> =A0 =A0 96 >> =A0 =A0 300 >> =A0 >> =A0 >> >> =A0 =A06 >> =A0 >> =A0 3 >> =A0 600 >> >> >> >> By doing this, I am able to see different pools per user, without >> mentioning anything inside the jobs. >> Automatically jobs are going to the respective pools. >> >> But what I wanted to know , is this the right method to do ? >> >> Thanks, >> Praveenesh >> >> >> >> On Wed, Jan 25, 2012 at 7:36 PM, Harsh J wrote: >> >>> Set the property in Pig with the 'set' command or other ways: >>> http://pig.apache.org/docs/r0.9.1/cmds.html#set or >>> http://pig.apache.org/docs/r0.9.1/start.html#properties >>> >>> As Srinivas covered earlier, pool allocation can be done per-user if >>> you set the scheduler poolnameproperty to "user.name". Per group if >>> you set the property to "group.name". >>> >>> Then you can provide per-poolname config overrides via the "pool" >>> element config described in >>> >>> http://hadoop.apache.org/common/docs/current/fair_scheduler.html#Alloca= tion+File+%28fair-scheduler.xml%29 >>> >>> On Wed, Jan 25, 2012 at 7:01 PM, praveenesh kumar >>> wrote: >>> > I am running pig jobs, how can I specify on which pool, it should run= ? >>> > Also do you mean, the pool allocation is done job wise, not user wise= ? >>> > >>> > >>> > On Wed, Jan 25, 2012 at 6:14 PM, Srinivas Surasani >>> wrote: >>> > >>> >> Praveenesh, >>> >> >>> >> You can try specifying "mapred.fairscheduler.pool" to your pool name >>> while >>> >> running the job. By default, mapred.faircheduler.poolnameproperty se= t >>> to >>> >> user.name ( each job run by user is allocated to his named pool ) an= d >>> you >>> >> can also change this property to group.name. >>> >> >>> >> Srinivas -- >>> >> >>> >> Also, you can set >>> >> >>> >> On Wed, Jan 25, 2012 at 6:24 AM, praveenesh kumar < >>> praveenesh@gmail.com >>> >> >wrote: >>> >> >>> >> > Understanding Fair Schedulers better. >>> >> > >>> >> > Can we create mulitple pools in Fair Schedulers. I guess Yes. Plea= se >>> >> > correct me. >>> >> > >>> >> > Suppose I have 2 pools in my fair-scheduler.xml >>> >> > >>> >> > 1. Hadoop-users : Min map : 10, Max map : 50, Min Reduce : 10, Max >>> >> Reduce : >>> >> > 50 >>> >> > 2. Admin-users: Min map : 20, Max map : 80, Min Reduce : 20, Max >>> Reduce : >>> >> > 80 >>> >> > >>> >> > I have 5 users, who will be using these pools. How will I allocate >>> >> specific >>> >> > pools to specific users ? >>> >> > >>> >> > Suppose I want user1,user2 to use "Hadoop-users" pool and >>> >> user3,user4,user5 >>> >> > to use "Admin users" >>> >> > >>> >> > In >>> http://hadoop.apache.org/common/docs/r0.20.205.0/fair_scheduler.html >>> >> > they have mentioned allocations something like this. >>> >> > >>> >> > >>> >> > >>> >> > =A0 >>> >> > =A0 =A05 >>> >> > =A0 =A05 >>> >> > =A0 =A025 >>> >> > =A0 =A025 >>> >> > =A0 =A0300 >>> >> > =A0 >>> >> > =A0 >>> >> > =A0 =A06 >>> >> > =A0 >>> >> > =A03 >>> >> > =A0600 >>> >> > >>> >> > >>> >> > I tried creating more pools, its happening, but how to allocate >>> users to >>> >> > use specific pools ? >>> >> > >>> >> > Thanks, >>> >> > Praveenesh >>> >> > >>> >> >>> >>> >>> >>> -- >>> Harsh J >>> Customer Ops. Engineer, Cloudera >>> >> >> --=20 Harsh J Customer Ops. Engineer, Cloudera