Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 89C1610DEF for ; Tue, 26 Nov 2013 23:01:56 +0000 (UTC) Received: (qmail 58385 invoked by uid 500); 26 Nov 2013 23:01:51 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 58287 invoked by uid 500); 26 Nov 2013 23:01:50 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 58280 invoked by uid 99); 26 Nov 2013 23:01:50 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Nov 2013 23:01:50 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=HTML_FONT_FACE_BAD,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of orenault@hortonworks.com designates 209.85.220.175 as permitted sender) Received: from [209.85.220.175] (HELO mail-vc0-f175.google.com) (209.85.220.175) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Nov 2013 23:01:46 +0000 Received: by mail-vc0-f175.google.com with SMTP id ld13so4265834vcb.6 for ; Tue, 26 Nov 2013 15:01:26 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=Ri0vOLrlEPOzSrhUmK7sMSXuutfq6HGl3LTOutCjF34=; b=SxS1qwT4zBI3kgqEmo1AGmcDZ2JLnHIrwJCoYz2AQeOsRCZrUhH41HC6o9EkI6V/fJ pgAOCZ0JURL1VoGKWaz85cwh6dzWglm/KGW1n3UXNkezcFTY6Jx0CsRXYPhCT9MoFi8w hbirDtl3OkoPzRDQBFhxHl9qNfv1XP525+W/KHxqd0EbDc/apzCNxAK5v+LxLUwupSbU V6PnY6A/uNcz322rnKQVkWwnGSR8UH6pdxmPpdnl5o1/50ntXpBnh92aWBUF0Ldw+Ksu zw7oYMrfb04Clrb8TI7J8VqMD5t3uRdZAdYxi6s1On5qaSre96C5xevi2IpWL+LJu0i0 k22Q== X-Gm-Message-State: ALoCoQkt1l2HqTuL4BNWro6lr5Q1U8PTbu2rcuZteartkBRJcGYJi/P1K2jxcIm29faz+8ViShT8O+uEPmDHLZzawaWQwAwo8SqyO3IKnhfBA3VaguYl0bY= MIME-Version: 1.0 X-Received: by 10.58.188.42 with SMTP id fx10mr693419vec.51.1385506885829; Tue, 26 Nov 2013 15:01:25 -0800 (PST) Received: by 10.58.34.135 with HTTP; Tue, 26 Nov 2013 15:01:25 -0800 (PST) In-Reply-To: References: Date: Tue, 26 Nov 2013 23:01:25 +0000 Message-ID: Subject: Re: Working with Capacity Scheduler From: Olivier Renault To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=089e013a27a6c3a03f04ec1c75d6 X-Virus-Checked: Checked by ClamAV on apache.org --089e013a27a6c3a03f04ec1c75d6 Content-Type: text/plain; charset=US-ASCII I don't believe the distro should matter. Could you confirm that you've got the following in capacity-scheduler.xml ? ( The permission are hierarchical, so you need to restrict the root queue ) yarn.scheduler.capacity.root.acl_submit_applications yarn.scheduler.capacity.root.acl_administer_queue Thanks, Olivier On 26 November 2013 22:51, Munna wrote: > Hi Olivier, > > "yarn.acl.enable" is enabled earlier, for your information i am using > Cloudera Manager to manage the cluster. again same problem :( > > > > > On Wed, Nov 27, 2013 at 3:37 AM, Olivier Renault > wrote: > >> Here is a working configuration. >> >> In yarn-site.xml, you'll need to enable >> yarn.acl.enable >> true >> >> Thanks, >> Olivier >> >> >> On 26 November 2013 21:48, Munna wrote: >> >>> Hi, >>> >>> Can i able to get the solution on Capacity Scheduler ? >>> >>> >>> On Wed, Nov 27, 2013 at 2:24 AM, Munna wrote: >>> >>>> yes... acl's are not enforced. >>>> >>>> >>>> On Wed, Nov 27, 2013 at 2:16 AM, Olivier Renault < >>>> orenault@hortonworks.com> wrote: >>>> >>>>> Do you mean that acl are not being enforced? >>>>> >>>>> Olivier >>>>> On 26 Nov 2013 20:18, "Munna" wrote: >>>>> >>>>>> Hi Olivier, >>>>>> >>>>>> have u hold any solution, why user are not identifying their queues? >>>>>> >>>>>> Thanks >>>>>> Munna >>>>>> >>>>>> >>>>>> On Tue, Nov 26, 2013 at 11:17 PM, Munna wrote: >>>>>> >>>>>>> *mapred queue -list* result: >>>>>>> >>>>>>> [user@host ~]$ mapred queue -list >>>>>>> 13/11/26 09:37:48 INFO service.AbstractService: >>>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. >>>>>>> 13/11/26 09:37:48 INFO service.AbstractService: >>>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. >>>>>>> ====================== >>>>>>> Queue Name : exploration >>>>>>> Queue State : running >>>>>>> Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0, >>>>>>> CurrentCapacity: 0.0 >>>>>>> ====================== >>>>>>> Queue Name : a >>>>>>> Queue State : running >>>>>>> Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0, >>>>>>> CurrentCapacity: 0.0 >>>>>>> ====================== >>>>>>> Queue Name : b >>>>>>> Queue State : running >>>>>>> Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0, >>>>>>> CurrentCapacity: 0.0 >>>>>>> ====================== >>>>>>> Queue Name : c >>>>>>> Queue State : running >>>>>>> Scheduling Info : Capacity: 40.0, MaximumCapacity: 1.0, >>>>>>> CurrentCapacity: 0.0 >>>>>>> ====================== >>>>>>> Queue Name : production >>>>>>> Queue State : running >>>>>>> Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, >>>>>>> CurrentCapacity: 0.0 >>>>>>> >>>>>>> and I saw in RM logs, mapred user try to find default queue, logs >>>>>>> are below: >>>>>>> >>>>>>> 2013-11-26 05:22:59,804 INFO >>>>>>> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application >>>>>>> with id 1 submitted by user mapred >>>>>>> 2013-11-26 05:22:59,808 INFO >>>>>>> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=mapred >>>>>>> IP=10.10.10.1 OPERATION=Submit Application Request >>>>>>> TARGET=ClientRMService RESULT=SUCCESS APPID=application_1385472086605_0001 >>>>>>> 2013-11-26 05:22:59,828 INFO >>>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: >>>>>>> application_1385472086605_0001 State change from NEW to SUBMITTED >>>>>>> 2013-11-26 05:22:59,829 INFO >>>>>>> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: >>>>>>> Registering appattempt_1385472086605_0001_000001 >>>>>>> 2013-11-26 05:22:59,830 INFO >>>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: >>>>>>> appattempt_1385472086605_0001_000001 State change from NEW to SUBMITTED >>>>>>> 2013-11-26 05:22:59,832 INFO >>>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: >>>>>>> appattempt_1385472086605_0001_000001 State change from SUBMITTED to FAILED >>>>>>> 2013-11-26 05:22:59,836 INFO >>>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: >>>>>>> application_1385472086605_0001 State change from SUBMITTED to FAILED >>>>>>> 2013-11-26 05:22:59,837 WARN >>>>>>> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=mapred >>>>>>> OPERATION=Application Finished - Failed TARGET=RMAppManager >>>>>>> RESULT=FAILURE DESCRIPTION=App failed with state: FAILED >>>>>>> PERMISSIONS=Application appattempt_1385472086605_0001_000001 submitted by >>>>>>> user mapred to unknown queue: default APPID=application_1385472086605_0001 >>>>>>> 2013-11-26 05:22:59,841 INFO >>>>>>> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: >>>>>>> Removing info for app: application_1385472086605_0001 >>>>>>> 2013-11-26 05:22:59,842 INFO >>>>>>> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: >>>>>>> appId=application_1385472086605_0001,name=select count(*) from >>>>>>> crc_exchange_rate(Stage-1),user=mapred,queue=default,state=FAILED,trackingUrl=hostname:8088/proxy/application_1385472086605_0001/,appMasterHost=N/A,startTime=1385472179799,finishTime=1385472179836 >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Nov 26, 2013 at 10:36 PM, Olivier Renault < >>>>>>> orenault@hortonworks.com> wrote: >>>>>>> >>>>>>>> Could you confirm that the queue are running by: >>>>>>>> mapred queue -list >>>>>>>> >>>>>>>> Otherwise what error do you get when submitting a job ? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Olivier >>>>>>>> >>>>>>>> >>>>>>>> On 26 November 2013 15:33, Munna wrote: >>>>>>>> >>>>>>>>> Below configured queue's are showing, then why i cannt able to run >>>>>>>>> the job. >>>>>>>>> >>>>>>>>> [root@host ~]# hadoop queue -showacls >>>>>>>>> DEPRECATED: Use of this script to execute mapred command is >>>>>>>>> deprecated. >>>>>>>>> Instead use the mapred command for it. >>>>>>>>> >>>>>>>>> 13/11/26 05:25:43 INFO service.AbstractService: >>>>>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. >>>>>>>>> 13/11/26 05:25:44 INFO service.AbstractService: >>>>>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. >>>>>>>>> Queue acls for user : root >>>>>>>>> >>>>>>>>> Queue Operations >>>>>>>>> ===================== >>>>>>>>> root ADMINISTER_QUEUE,SUBMIT_APPLICATIONS >>>>>>>>> exploration ADMINISTER_QUEUE,SUBMIT_APPLICATIONS >>>>>>>>> a ADMINISTER_QUEUE,SUBMIT_APPLICATIONS >>>>>>>>> b ADMINISTER_QUEUE,SUBMIT_APPLICATIONS >>>>>>>>> c ADMINISTER_QUEUE,SUBMIT_APPLICATIONS >>>>>>>>> production ADMINISTER_QUEUE,SUBMIT_APPLICATIONS >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Nov 26, 2013 at 8:05 PM, Olivier Renault < >>>>>>>>> orenault@hortonworks.com> wrote: >>>>>>>>> >>>>>>>>>> Sorry my mistake, it's : >>>>>>>>>> mapred queue -showacls >>>>>>>>>> >>>>>>>>>> Or you can also use hadoop queue -showacls as suggested by >>>>>>>>>> Jitendra >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Olviier >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 26 November 2013 14:17, Munna wrote: >>>>>>>>>> >>>>>>>>>>> Yes! I have executed same command: >>>>>>>>>>> >>>>>>>>>>> [user@host ~]$* yarn queue -showacls* >>>>>>>>>>> >>>>>>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError: queue >>>>>>>>>>> Caused by: java.lang.ClassNotFoundException: queue >>>>>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >>>>>>>>>>> at java.security.AccessController.doPrivileged(Native >>>>>>>>>>> Method) >>>>>>>>>>> at >>>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:190) >>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >>>>>>>>>>> at >>>>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >>>>>>>>>>> Could not find the main class: queue. Program will exit. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Nov 26, 2013 at 7:25 PM, Olivier Renault < >>>>>>>>>>> orenault@hortonworks.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> it should be : >>>>>>>>>>>> yarn queue -showacls >>>>>>>>>>>> >>>>>>>>>>>> Olivier >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 26 November 2013 13:28, Munna wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Olivier, >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you for your reply. >>>>>>>>>>>>> >>>>>>>>>>>>> As you said, i ran those commands and i am getting following >>>>>>>>>>>>> error message on both the commands. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> *[root@host ~]# sudo -u yarn yarn queue -showacls Exception in >>>>>>>>>>>>> thread "main" java.lang.NoClassDefFoundError: queueCaused by: >>>>>>>>>>>>> java.lang.ClassNotFoundException: queue at >>>>>>>>>>>>> java.net.URLClassLoader$1.run(URLClassLoader.java:202) at >>>>>>>>>>>>> java.security.AccessController.doPrivileged(Native Method) at >>>>>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:190) at >>>>>>>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:306) at >>>>>>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at >>>>>>>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:247) Could not find the >>>>>>>>>>>>> main class: queue. Program will exit.* >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Munna >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Nov 26, 2013 at 4:10 PM, Olivier Renault < >>>>>>>>>>>>> orenault@hortonworks.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Could you maybe send us the output of : >>>>>>>>>>>>>> - yarn queue -showacls >>>>>>>>>>>>>> - yarn queue -list >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Olivier >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 26 November 2013 05:53, Munna wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I working with Capacity Scheduler on YARN and I have >>>>>>>>>>>>>>> configured different queues. I can able to see all the queues on RM ui. >>>>>>>>>>>>>>> But, when i start to run MR jobs with configured user names(yarn,mapred), i >>>>>>>>>>>>>>> am unable to run the Jobs and job are suspended. Again i set default as >>>>>>>>>>>>>>> FIFO working fine. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Can you please help me out sort this issue and configured >>>>>>>>>>>>>>> Configurations are given below. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> * >>>>>>>>>>>>>>> yarn.scheduler.capacity.root.queues >>>>>>>>>>>>>>> production,exploration >>>>>>>>>>>>>>> yarn.scheduler.capacity.root.exploration.queues >>>>>>>>>>>>>>> a,b,c >>>>>>>>>>>>>>> yarn.scheduler.capacity.root.capacity >>>>>>>>>>>>>>> 100 >>>>>>>>>>>>>>> yarn.scheduler.capacity.root.production.capacity >>>>>>>>>>>>>>> 70 >>>>>>>>>>>>>>> yarn.scheduler.capacity.root.exploration.capacity >>>>>>>>>>>>>>> 30 >>>>>>>>>>>>>>> yarn.scheduler.capacity.root.exploration.a.capacity >>>>>>>>>>>>>>> 30 >>>>>>>>>>>>>>> yarn.scheduler.capacity.root.exploration.b.capacity >>>>>>>>>>>>>>> 30 >>>>>>>>>>>>>>> yarn.scheduler.capacity.root.exploration.c.capacity >>>>>>>>>>>>>>> 40 yarn.scheduler.capacity.root. >>>>>>>>>>>>>>> production.acl_submit_applications >>>>>>>>>>>>>>> yarn,mapred userb >>>>>>>>>>>>>>> yarn.scheduler.capacity.root.exploration.a.acl_submit_applications >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> yarn.scheduler.capacity.root.exploration.b.acl_submit_applications >>>>>>>>>>>>>>> userb >>>>>>>>>>>>>>> yarn.scheduler.capacity.root.exploration.c.acl_submit_applications >>>>>>>>>>>>>>> userc* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> *Regards* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *Munna* >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>> CONFIDENTIALITY NOTICE >>>>>>>> NOTICE: This message is intended for the use of the individual or >>>>>>>> entity to which it is addressed and may contain information that is >>>>>>>> confidential, privileged and exempt from disclosure under applicable law. >>>>>>>> If the reader of this message is not the intended recipient, you are hereby >>>>>>>> notified that any printing, copying, dissemination, distribution, >>>>>>>> disclosure or forwarding of this communication is strictly prohibited. If >>>>>>>> you have received this communication in error, please contact the sender >>>>>>>> immediately and delete it from your system. Thank You. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Regards* >>>>>>> >>>>>>> *Munna* >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Regards* >>>>>> >>>>>> *Munna* >>>>>> >>>>> >>>>> CONFIDENTIALITY NOTICE >>>>> NOTICE: This message is intended for the use of the individual or >>>>> entity to which it is addressed and may contain information that is >>>>> confidential, privileged and exempt from disclosure under applicable law. >>>>> If the reader of this message is not the intended recipient, you are hereby >>>>> notified that any printing, copying, dissemination, distribution, >>>>> disclosure or forwarding of this communication is strictly prohibited. If >>>>> you have received this communication in error, please contact the sender >>>>> immediately and delete it from your system. Thank You. >>>>> >>>> >>>> >>>> >>>> -- >>>> *Regards* >>>> >>>> *Munna* >>>> >>> >>> >>> >>> -- >>> *Regards* >>> >>> *Munna* >>> >> >> >> >> -- >> * Olivier Renault * Solution Engineer >> ------------------------------ >> >> Phone: +44 7500 933 036 >> Email: orenault@hortonworks.com >> Website: http://www.hortonworks.com/ >> >> * Follow Us: * >> >> >> >> >> [image: photo] >> >> Latest From Our Blog: SAP HANA + Hadoop: A Perfect Match >> >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity >> to which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader >> of this message is not the intended recipient, you are hereby notified that >> any printing, copying, dissemination, distribution, disclosure or >> forwarding of this communication is strictly prohibited. If you have >> received this communication in error, please contact the sender immediately >> and delete it from your system. Thank You. >> > > > > -- > *Regards* > > *Munna* > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. --089e013a27a6c3a03f04ec1c75d6 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
I don't believe the distro should matter.=A0

<= /div>
Could you confirm that you've got the following in capacity-s= cheduler.xml ? ( The permission are hierarchical, so you need to restrict t= he root queue )
=A0 <property>
=A0 =A0 =A0 <name>yarn.sched= uler.capacity.root.acl_submit_applications</name>
=A0 =A0 = =A0 <value> </value>
=A0 </property>
=
=A0 <property>
=A0 =A0 =A0 <name>yarn.scheduler.capac= ity.root.acl_administer_queue</name>
=A0 =A0 =A0 <value&= gt; </value>

Thanks,=A0
Olivier

On 26 = November 2013 22:51, Munna <munnavarsk@gmail.com> wrote:<= br>
Hi Olivier,

"yarn.acl.enable"=A0 is = enabled earlier, for your information i am using Cloudera Manager to manage= the cluster. again same problem :(

=A0


On Wed, Nov 27, 2013 at 3:37 AM, Olivier= Renault <orenault@hortonworks.com> wrote:
Here is a working configuration.=A0

In = yarn-site.xml, you'll need to enable=A0
=A0 =A0 <name>ya= rn.acl.enable</name>
=A0 =A0 <value>true</value>= ;

Thanks,
Olivier


On 26 Novembe= r 2013 21:48, Munna <munnavarsk@gmail.com> wrote:
Hi,

Can i able to get the sol= ution on Capacity Scheduler ?


On Wed, Nov 27, 2013 at 2:24 A= M, Munna <munnavarsk@gmail.com> wrote:
yes... acl's are not enforced.
<= div class=3D"gmail_extra">


On Wed, Nov 27, 2013 at 2:16 AM, Olivier Renault <orenault@hortonwo= rks.com> wrote:

Do you mean that acl are not being enforced= ?

Olivier

On 26 Nov 2013 20:18, "Munna" <munnavarsk@gmail.com<= /a>> wrote:
Hi Olivier,

have u hold any solutio= n, why user are not identifying their queues?

Thanks
<= /div>Munna


On Tue, Nov 26, 2013 at 11:17 PM, Munna <munnavarsk@gmail.com> wrote:
mapred queue -list result:

[user@host ~= ]$ mapred queue -list
13/11/26 09:37:48 INFO service.AbstractService: Se= rvice:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
13/11/26 0= 9:37:48 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client= .YarnClientImpl is started.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Queue= Name : exploration
Queue State : running
Scheduling Info : Capacity:= 30.000002, MaximumCapacity: 1.0, CurrentCapacity: 0.0
=A0=A0=A0 =3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
=A0=A0=A0 Q= ueue Name : a
=A0=A0=A0 Queue State : running
=A0=A0=A0 Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0, Curr= entCapacity: 0.0
=A0=A0=A0 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D
=A0=A0=A0 Queue Name : b
=A0=A0=A0 Queue State = : running
=A0=A0=A0 Scheduling Info : Capacity: 30.000002, MaximumCapaci= ty: 1.0, CurrentCapacity: 0.0
=A0=A0=A0 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D
=A0=A0=A0 Queue Name : c
=A0=A0=A0 Queue State : running
=A0= =A0=A0 Scheduling Info : Capacity: 40.0, MaximumCapacity: 1.0, CurrentCapac= ity: 0.0
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D
Queue Name : production
Queue State : running
Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, CurrentCapacity: 0.= 0

and I saw in RM logs, mapred user try to find default queue,= logs are below:

2013-11-26 05:22:59,804 INFO org.apache.hadoop.yarn= .server.resourcemanager.ClientRMService: Application with id 1 submitted by= user mapred
2013-11-26 05:22:59,808 INFO org.apache.hadoop.yarn.server.resourcemanager.= RMAuditLogger: USER=3Dmapred=A0=A0 IP=3D10.10.10.1=A0 OPERATION=3DSubmit Ap= plication Request=A0=A0=A0 TARGET=3DClientRMService=A0 RESULT=3DSUCCESS=A0 = APPID=3Dapplication_1385472086605_0001
2013-11-26 05:22:59,828 INFO org.apache.hadoop.yarn.server.resourcemanager.= rmapp.RMAppImpl: application_1385472086605_0001 State change from NEW to SU= BMITTED
2013-11-26 05:22:59,829 INFO org.apache.hadoop.yarn.server.resou= rcemanager.ApplicationMasterService: Registering appattempt_1385472086605_0= 001_000001
2013-11-26 05:22:59,830 INFO org.apache.hadoop.yarn.server.resourcemanager.= rmapp.attempt.RMAppAttemptImpl: appattempt_1385472086605_0001_000001 State = change from NEW to SUBMITTED
2013-11-26 05:22:59,832 INFO org.apache.had= oop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_= 1385472086605_0001_000001 State change from SUBMITTED to FAILED
2013-11-26 05:22:59,836 INFO org.apache.hadoop.yarn.server.resourcemanager.= rmapp.RMAppImpl: application_1385472086605_0001 State change from SUBMITTED= to FAILED
2013-11-26 05:22:59,837 WARN org.apache.hadoop.yarn.server.re= sourcemanager.RMAuditLogger: USER=3Dmapred=A0=A0 OPERATION=3DApplication Fi= nished - Failed TARGET=3DRMAppManager=A0=A0=A0=A0 RESULT=3DFAILURE=A0 DESCR= IPTION=3DApp failed with state: FAILED=A0=A0=A0=A0=A0=A0 PERMISSIONS=3DAppl= ication appattempt_1385472086605_0001_000001 submitted by user mapred to un= known queue: default APPID=3Dapplication_1385472086605_0001
2013-11-26 05:22:59,841 INFO org.apache.hadoop.yarn.server.resourcemanager.= recovery.RMStateStore: Removing info for app: application_1385472086605_000= 1
2013-11-26 05:22:59,842 INFO org.apache.hadoop.yarn.server.resourceman= ager.RMAppManager$ApplicationSummary: appId=3Dapplication_1385472086605_000= 1,name=3Dselect count(*) from crc_exchange_rate(Stage-1),user=3Dmapred,queu= e=3Ddefault,state=3DFAILED,trackingUrl=3Dhostname:8088/proxy/application_13= 85472086605_0001/,appMasterHost=3DN/A,startTime=3D1385472179799,finishTime= =3D1385472179836



On Tue, Nov 26, 2013 at 10:36 PM, Olivier Renault = <orenault@= hortonworks.com> wrote:
Could you confirm that the queue are runn= ing by:
mapred queue -list

Otherwise what error do you get= when submitting a job ?

Thanks,=A0
Oliv= ier


On 26 Novembe= r 2013 15:33, Munna <munnavarsk@gmail.com> wrote:
Below configured queue's are showing, then why i cannt= able to run the job.

[root@host ~]# hadoop queue -showacls
DEPRE= CATED: Use of this script to execute mapred command is deprecated.
Inste= ad use the mapred command for it.

13/11/26 05:25:43 INFO service.AbstractService: Service:org.apache.hado= op.yarn.client.YarnClientImpl is inited.
13/11/26 05:25:44 INFO service.= AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is st= arted.
Queue acls for user :=A0 root

Queue=A0 Operations
=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
root=A0 ADMINISTER_QUEU= E,SUBMIT_APPLICATIONS
exploration=A0 ADMINISTER_QUEUE,SUBMIT_APPLICATION= S
a=A0 ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
b=A0 ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
c=A0 ADMINISTER_QUEUE,SUBMIT_A= PPLICATIONS
production=A0 ADMINISTER_QUEUE,SUBMIT_APPLICATIONS



On Tue, Nov 26, 2013 at 8:05 PM, Olivier Renault <orenault@hortonwo= rks.com> wrote:
Sorry my mistake, it's :=A0
mapre= d queue -showacls

Or you can also use hadoop queue -showacls as suggested= by=A0= Jitendra

Thanks,=A0<= /div>
Olviier


On 26 November 2013 14:17, Munna <munnavarsk@gmail.com> wrote:
Yes! I have executed same command:
[user@host ~]$ yarn queue -showacls

Exception in thread "main" java.lang.NoClassDefFoundError: qu= eue
Caused by: java.lang.ClassNotFoundException: queue
=A0=A0=A0=A0=A0=A0=A0 at java.net.URLClassLoader$1.run(URLClassLoader.java:= 202)
=A0=A0=A0=A0=A0=A0=A0 at java.security.AccessController.doPrivilege= d(Native Method)
=A0=A0=A0=A0=A0=A0=A0 at java.net.URLClassLoader.findCl= ass(URLClassLoader.java:190)
=A0=A0=A0=A0=A0=A0=A0 at java.lang.ClassLoa= der.loadClass(ClassLoader.java:306)
=A0=A0=A0=A0=A0=A0=A0 at sun.misc.Launcher$AppClassLoader.loadClass(Launche= r.java:301)
=A0=A0=A0=A0=A0=A0=A0 at java.lang.ClassLoader.loadClass(Cla= ssLoader.java:247)
Could not find the main class: queue.=A0 Program will= exit.



On Tue, Nov 26, 2013 at 7:25 PM, Olivier= Renault <orenault@hortonworks.com> wrote:
it should be :
yarn queue -showacls

Olivier
=


On 26 November 2013 13:28, Munna <munn= avarsk@gmail.com> wrote:
Hi Olivier,

Thank you for your reply.

As you said, i ran those commands an= d i am getting following error message on both the commands.

[root@host ~]# sudo -u yarn yarn queue -showacls
Exception in thread "main" java.lang.NoClassDefFoundError: queue<= br>Caused by: java.lang.ClassNotFoundException: queue
=A0=A0=A0=A0=A0=A0= =A0 at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
=A0=A0=A0= =A0=A0=A0=A0 at java.security.AccessController.doPrivileged(Native Method)<= br> =A0=A0=A0=A0=A0=A0=A0 at java.net.URLClassLoader.findClass(URLClassLoader.j= ava:190)
=A0=A0=A0=A0=A0=A0=A0 at java.lang.ClassLoader.loadClass(ClassL= oader.java:306)
=A0=A0=A0=A0=A0=A0=A0 at sun.misc.Launcher$AppClassLoade= r.loadClass(Launcher.java:301)
=A0=A0=A0=A0=A0=A0=A0 at java.lang.ClassL= oader.loadClass(ClassLoader.java:247)
Could not find the main class: queue.=A0 Program will exit.

Regards,
Munna



On Tue, Nov 26, 2013 at 4:10 PM, Olivier Renault <orenault@hortonwo= rks.com> wrote:
Could you maybe send us the o= utput of :
=A0- yarn queue -showacls
=A0- yarn queue -list

Thanks,=A0
Olivier


On 26 November 2013 05:53, Munna <munnavarsk@gmail.com> w= rote:
Hi,

I working with Capacity Schedul= er on YARN and I have configured different queues. I can able to see all th= e queues on RM ui. But, when i start to run MR jobs with configured user na= mes(yarn,mapred), i am unable to run the Jobs and job are suspended. Again = i set default as FIFO working fine.

Can you please help me out sort this issue and configured Configu= rations are given below.

<property>
=A0=A0=A0 <name&g= t;yarn.scheduler.capacity.root.queues</name>
=A0=A0=A0 <value&g= t;production,exploration</value>
</property>
<property>
=A0=A0=A0 <name>yarn.schedul= er.capacity.root.exploration.queues</name>
=A0=A0=A0 <value>= a,b,c</value>
</property>

<property>
=A0=A0= =A0 <name>yarn.scheduler.capacity.root.capacity</name>
=A0=A0=A0 <value>100</value>
</property>

<pr= operty>
=A0=A0=A0 <name>yarn.scheduler.capacity.root.production= .capacity</name>
=A0=A0=A0 <value>70</value>
</p= roperty>
<property>
=A0=A0=A0 <name>yarn.scheduler.capacity.root.expl= oration.capacity</name>
=A0=A0=A0 <value>30</value></property>

<property>
=A0=A0=A0 <name>yarn.sc= heduler.capacity.root.exploration.a.capacity</name>
=A0=A0=A0 <value>30</value>
</property>

<pro= perty>
=A0=A0=A0 <name>yarn.scheduler.capacity.root.exploration= .b.capacity</name>
=A0=A0=A0 <value>30</value>
<= /property>

<property>
=A0=A0=A0 <name>yarn.scheduler.capacity.root.= exploration.c.capacity</name>
=A0=A0=A0 <value>40</value&= gt;
</property>


<property>
=A0 <name>yar= n.scheduler.capacity.root. production.acl_submit_applications</name><= br> =A0 <value>yarn,mapred</value>
</property>

<= property>
=A0=A0=A0 <value>userb</value>
=A0=A0=A0 <= ;name>yarn.scheduler.capacity.root.exploration.a.acl_submit_applications= </name>
</property>

<property>
=A0=A0=A0 <name>yarn.sch= eduler.capacity.root.exploration.b.acl_submit_applications</name>
= =A0=A0=A0 <value>userb</value>
</property>

<= property>
=A0=A0=A0 <name>yarn.scheduler.capacity.root.exploration.c.acl_submit= _applications</name>
=A0=A0=A0 <value>userc</value></property>

</configuration>


--
Regards

Munna



<= /blockquote>
<= /div>

CONFIDENTIALITY NOTICE
NOTICE: This message is = intended for the use of the individual or entity to which it is addressed a= nd may contain information that is confidential, privileged and exempt from= disclosure under applicable law. If the reader of this message is not the = intended recipient, you are hereby notified that any printing, copying, dis= semination, distribution, disclosure or forwarding of this communication is= strictly prohibited. If you have received this communication in error, ple= ase contact the sender immediately and delete it from your system. Thank Yo= u.



--
Regards

Munna<= /span>



--
Regards

Munna<= /span>

CONFIDENTIALITY NOTICE
NOTICE: This message is = intended for the use of the individual or entity to which it is addressed a= nd may contain information that is confidential, privileged and exempt from= disclosure under applicable law. If the reader of this message is not the = intended recipient, you are hereby notified that any printing, copying, dis= semination, distribution, disclosure or forwarding of this communication is= strictly prohibited. If you have received this communication in error, ple= ase contact the sender immediately and delete it from your system. Thank Yo= u.



--
Regards

Munna<= /span>


--
Regards

Munna<= /span>


--=
Olivier Renault =A0 Solution Engineer
=A0
Phone: =A0 =A0 =A0 +44 7500 933 036
Email:=A0 =A0 =A0 orenault@hortonworks.com
=A0
Follow Us:

3D"photo"


CONFIDENTIALITY NOTICE
NOTICE: This message is = intended for the use of the individual or entity to which it is addressed a= nd may contain information that is confidential, privileged and exempt from= disclosure under applicable law. If the reader of this message is not the = intended recipient, you are hereby notified that any printing, copying, dis= semination, distribution, disclosure or forwarding of this communication is= strictly prohibited. If you have received this communication in error, ple= ase contact the sender immediately and delete it from your system. Thank Yo= u.



--
Regards<= div style>
Munna

CONFIDENTIALITY NOTICE
NOTICE: This message is = intended for the use of the individual or entity to which it is addressed a= nd may contain information that is confidential, privileged and exempt from= disclosure under applicable law. If the reader of this message is not the = intended recipient, you are hereby notified that any printing, copying, dis= semination, distribution, disclosure or forwarding of this communication is= strictly prohibited. If you have received this communication in error, ple= ase contact the sender immediately and delete it from your system. Thank Yo= u. --089e013a27a6c3a03f04ec1c75d6--