Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8BC06107C8 for ; Thu, 21 Nov 2013 19:24:14 +0000 (UTC) Received: (qmail 84757 invoked by uid 500); 21 Nov 2013 19:24:09 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 84589 invoked by uid 500); 21 Nov 2013 19:24:09 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 84582 invoked by uid 99); 21 Nov 2013 19:24:09 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Nov 2013 19:24:09 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [98.138.91.118] (HELO nm25-vm6.bullet.mail.ne1.yahoo.com) (98.138.91.118) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Nov 2013 19:24:00 +0000 Received: from [98.138.100.118] by nm25.bullet.mail.ne1.yahoo.com with NNFMP; 21 Nov 2013 19:23:39 -0000 Received: from [98.138.89.250] by tm109.bullet.mail.ne1.yahoo.com with NNFMP; 21 Nov 2013 19:23:39 -0000 Received: from [127.0.0.1] by omp1042.mail.ne1.yahoo.com with NNFMP; 21 Nov 2013 19:23:39 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 139245.83179.bm@omp1042.mail.ne1.yahoo.com Received: (qmail 79526 invoked by uid 60001); 21 Nov 2013 19:23:38 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1385061818; bh=B8sEOQL/vb3Ijm9bE9gAGI7+Jra2gqYW9g4y9DVKT9I=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=H6XDcArlv6L6AutL21gosoS8JFdcX/IJvRUEEmXWG9gQqdmMKl9nWhGbfZthVg2cA/vTjh/E1RTjScoHE3VL6FbzwlNR/f1p8nFjaq7kkhuRkxlBJtB05CwV/V195Ex9Ch/6hzESo9AZyYXQU8GNL2TitC75n/oEEM3tTQgaNRY= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=f8GXyoLm3IE4KE8Ps+/Kg5cXhC+NyOWSL7QfweOVL6g+fJcSab3oFxG1IYifMrQiOEWx7pfCkwOvccnRmVqiZOpIjdJLzz1imePTklaHffL5ZMDZFTzaSdnLXDxljsUYDWkT7vnLfJ59MIVxVr1+W1ElTPC09r8hihR7zzQ6/ww=; X-YMail-OSG: p3._isUVM1lNz8W6AdlxpIojksgeSr3ejkOsG1ro5RXEHp5 BYEQrsVA7ni_W9lRYpmGw8ml2EHXsdgXrxtyiZXPl2M4NkOHD8N6Kq2rTogZ a2gYZIetBy5bQXdZD0EBCnJk1v9CIXfXvRnhDUnlpWp6fiiIpWka3GAUFpTi gYOU.HVzATMyKFAgm2gnPEVkLOw6TnNdF6ks7OnvY2By6VJgGS7mkEn7wZmz ooD2VC6DnjNRVcgKNrlPgWMUNW3hU3dmlInEfhs9VIwTzFA9jKFrDwDC1x4B jGczwODu9cgnUmn5K7OWLYbpocEHHCJaLmTvxw8m2E17tSZ5s9tse.3khBbn vdX7jcAcf26_FlJbWDHAM0YFNCXzBsPhhkz3axe_O_bRI8vp9UDrwb.xamt2 0b.cHdrNIzCrnAqQgSU.EK8XOglbf1BwTAIgC1jUREm458YEKjUPDaRMOh_o nc1y9ZWYDWsd2LgcNiCX8Wghg.NQEvfe8lSdq9k2h96.EGv9jSVsafL0mE8W FI1GZQfCQ7LneZTYt5A.4KEpYdApdravBORjDLyKOhabmuHuYDje3GxX.OMR QH2fPSUKDcdOgJ3wLcaLIbLPJmTiuBbqR5YJyJw5epOXTV.hzDDMUKVS_Y2D 9tXcgOxcaQwgZVVcN0MJDlFf6A01waNaJX4tZAzsLb6IJcB8- Received: from [64.125.69.70] by web161703.mail.bf1.yahoo.com via HTTP; Thu, 21 Nov 2013 11:23:38 PST X-Rocket-MIMEInfo: 002.001,SGVyZSdzIHN0ZXAtYnktc3RlcCBvbiBob3cgdG8gZG8gdGhpczoKPT09PT09PT09PT09PT09PQoKSGVyZSdzIHN0ZXAgYnkgc3RlcCBvbiBob3cgdG8gZG8gdGhpczoKCjEuIEFkZCBmb2xsb3dpbmcgaW4gbWFwcmVkLXNpdGUueG1sOgoKCsKgIDxwcm9wZXJ0eT4KwqDCoMKgIDxuYW1lPm1hcHJlZC5mYWlyc2NoZWR1bGVyLnByZWVtcHRpb248L25hbWU.CsKgwqDCoCA8ZmluYWw.dHJ1ZTwvZmluYWw.CsKgwqDCoCA8dmFsdWU.dHJ1ZTwvdmFsdWU.IArCoCA8L3Byb3BlcnR5PgrCoCA8cHJvcGVydHk.CsKgwqABMAEBAQE- X-Mailer: YahooMailWebService/0.8.166.601 References: <1385061649.86176.YahooMailNeo@web161706.mail.bf1.yahoo.com> Message-ID: <1385061818.78022.YahooMailNeo@web161703.mail.bf1.yahoo.com> Date: Thu, 21 Nov 2013 11:23:38 -0800 (PST) From: anurag tangri Reply-To: anurag tangri Subject: Re: Jobtracker fair scheduler To: "user@hadoop.apache.org" , anurag tangri In-Reply-To: <1385061649.86176.YahooMailNeo@web161706.mail.bf1.yahoo.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="-67045276-656092795-1385061818=:78022" X-Virus-Checked: Checked by ClamAV on apache.org ---67045276-656092795-1385061818=:78022 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Here's step-by-step on how to do this:=0A=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=0A=0AHere's step by step on how to do this:=0A=0A1. Add fol= lowing in mapred-site.xml:=0A=0A=0A=A0 =0A=A0=A0=A0 mapred.= fairscheduler.preemption=0A=A0=A0=A0 true=0A=A0=A0=A0= true =0A=A0 =0A=A0 =0A=A0=A0=A0 = mapred.jobtracker.taskScheduler=A0=A0 =0A=A0=A0=A0 true=0A=A0=A0=A0 org.apache.hadoop.mapred.FairScheduler=0A=A0 = =0A=A0 =0A=A0=A0=A0 mapred.acls.enabled= =0A=A0=A0=A0 true=0A=A0=A0=A0 true=0A=A0 =0A=A0=A0=A0 =0A=A0=A0=A0 mapred.fairscheduler.allow= .undeclared.pools=0A=A0=A0=A0 true=0A=A0=A0=A0 false=0A=A0 =0A=A0 =0A=A0=A0=A0 mapred.= fairscheduler.poolnameproperty=0A=A0=A0=A0 true=0A=A0= =A0=A0 mapred.job.queue.name=0A=A0 =0A=A0 =0A=A0=A0=A0 mapred.fairscheduler.allocation.file=0A=A0=A0= =A0 true=0A=A0=A0=A0 /etc/hadoop/conf/allocations.xml= =0A=A0 =0A=A0 =0A=A0=A0=A0 mapred.queue.= names=0A=A0=A0=A0 true=0A=A0=A0=A0 sqoop,defau= lt=0A=A0 =0A=A0 =0A=A0=A0=A0 mapreduce.j= ob.acl-view-job=0A=A0=A0=A0 true=0A=A0=A0=A0 *= =0A=A0 =0A=0A=0A2. Create mapred-queue-acls.xml in same = dir as mapred-site.xml where you define all the queues:=0A=0A=0A=A0 =0A=A0=A0=A0 mapred.queue.sqoop.acl-submit-job=0A=A0=A0=A0 usera, userb=0A=A0 =0A=A0 =0A=A0=A0=A0 mapred.queue.sqoop.acl-administer-jobs=0A=A0=A0= =A0 usera, userc=0A=A0 =0A=0A=0A=0A=0A= 3. Define fair scheduler allocations file (location specified in maped-site= .xml) to use the queues defined and assign resources to them:=0A=0A=0A=0A=A0 300= =0A=A0 =0A=A0=A0=A0= 700=0A=A0=A0=A0 175=0A=A0=A0= =A0 25=0A=A0 =0A=A0 =0A=A0=A0=A0 120=0A=A0=A0=A0 40=0A=A0=A0=A0 40=0A=A0 =0A= =A0 600=0A=0A=0A=0A4. After this, restart job tracker.=0A=0A=0AAnd your queues = should be in place. You can then submit job to right queue using =0A-Dmapre= d.job.queue.name=3Dsqoop or default =0A=0AAnd check the job going to actual= queue at the link:=0Ahttp://:50030/scheduler =0A=0A= =0ALook at the property mapreduce.job.acl-view-job in mapred-site. Means an= yone can view the jobs.=0A=0A=0ATry above and let us know any issues you st= ill face.=0A=0A=0A=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=0A=0A=0A=A0=0AAnurag Tangri=0A=0A= =0ANever wear your best trousers when you go out to fight for freedom and= =0Atruth.- Henrik Ibsen=0A=0A=0A=0AOn Thursday, November 21, 2013 11:21 AM,= anurag tangri wrote:=0A =0AHi Viswanathan,=0A=0A= What steps have you followed to set fair scheduler ?=0A=0AThanks,=0A=0A=A0A= nurag Tangri=0A=0A=0ANever wear your best trousers when you go out to fight= for freedom and=0Atruth.- Henrik Ibsen=0A=0A=0A=0AOn Thursday, November 21= , 2013 10:25 AM, Viswanathan J wrote:=0A =0AHi= ,=0AI'm running hadoop with 1.2.1, all my jobs are running in single queue = (Queue 1) only all the time. But I have configured default, queue 1&2.=0AWh= y jobs are not scheduled to all the queues.=0APlease help. Running like thi= s will be any issue?=0AThanks, ---67045276-656092795-1385061818=:78022 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable
Here's step-by-step on how to do this:
=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

Here's step by step on how to do this= :

1. Add following in mapred-site.xml:


  <propert= y>
    <name>mapred.fairscheduler.preemption<= /name>
    <final>true</final>
 &n= bsp;  <value>true</value>
  </property>
=   <property>
    <name>mapred.jobtracker= .taskScheduler</name>  
    <final>= ;true</final>
    <value>org.apache.hadoop.ma= pred.FairScheduler</value>
  </property>
  <= property>
    <name>mapred.acls.enabled</name= >
    <final>true</final>
    <value>true<= ;/value>
  </property>
    <property&= gt;
    <name>mapred.fairscheduler.allow.undeclared= .pools</name>
    <final>true</final>    <value>false</value>
  </proper= ty>
  <property>
    <name>mapred.= fairscheduler.poolnameproperty</name>
    <final= >true</final>
    <value>mapred.job.queue.= name</value>
  </property>
  <property>    <name>mapred.fairscheduler.allocation.file</n= ame>
    <final>true</final>
 &nbs= p;  <value>/etc/hadoop/conf/allocations.xml</value>
&nb= sp; </property>
  <property>
    <name>mapred.queue.names</= name>
    <final>true</final>
 &nb= sp;  <value>sqoop,default</value>
  </property&= gt;
  <property>
    <name>mapreduce.= job.acl-view-job</name>
    <final>true</f= inal>
    <value>*</value>
  </= property>


2. Create mapred-queue-acls.xml in same dir as mapr= ed-site.xml where you define all the queues:

<configuration>  <property>
    <name>mapred.queue.sq= oop.acl-submit-job</name>
    <value>usera, u= serb</value>
  </property>
  <property>    <name>mapred.queue.sqoop.acl-administer-jobs<= /name>
    <value>usera, userc</value>
  </property>
<! above two propert= ies for every queue defined in mapred-site.xml -->
</configuration= >


3. Define fair scheduler allocations file (location specifi= ed in maped-site.xml) to use the queues defined and assign resources to the= m:

<?xml version=3D"1.0"?>
<allocations>
  &l= t;defaultMinSharePreemptionTimeout>300</defaultMinSharePreemptionTime= out>
  <pool name=3D"sqoop">
    <min= Maps>700</minMaps>
    <minReduces>175<= /minReduces>
    <maxRunningJobs>25</maxRunni= ngJobs>
  </pool>
  <pool name=3D"default"><= br>    <minMaps>120</minMaps>
  &nb= sp; <minReduces>40</minReduces>
    <maxRu= nningJobs>40</maxRunningJobs>
  </pool>
  <fairSharePreemptionTimeout>600</fairSharePreemptionTimeout></allocations>


4. After this, restart job tracker.

And your queues should be in place. You can then submit job to right = queue using
-Dmapred.job.queue.name=3Dsqoop or default

And chec= k the job going to actual queue at the link:
http://<job tracker host= name>:50030/scheduler


Look at the property mapreduce.job.acl= -view-job in mapred-site. Means anyone can view the jobs.


Try ab= ove and let us know any issues you still face.


=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D

 
Anurag Tangri


Never wear your best trousers when you go out t= o fight for freedom and
truth.- Henrik Ibsen


On Thursday, Novembe= r 21, 2013 11:21 AM, anurag tangri <anurag_tangri@yahoo.com> wrote:
Hi Viswanathan,
What steps have you fol= lowed to set fair scheduler ?

Thanks,<= br clear=3D"none">
 Anurag Tangri


Never wear your best trousers when = you go out to fight for freedom and
truth.- Henrik Ibsen<= /div>


On Thursday, November 21, 2013 10:25 AM, Viswanathan= J <jayamviswanathan@gmail.com> wrote:
=
Hi,
=0A
I'm running hadoop with 1.2.1, all my jobs are runnin= g in single queue (Queue 1) only all the time. But I have configured defaul= t, queue 1&2.
=0A
Why jobs are not scheduled to all the queues= .
=0A
Please help. Running like this will be any issue?
=0AThanks,
=0A




---67045276-656092795-1385061818=:78022--