Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7E80910C5A for ; Tue, 26 Nov 2013 17:07:48 +0000 (UTC) Received: (qmail 11818 invoked by uid 500); 26 Nov 2013 17:07:05 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 11606 invoked by uid 500); 26 Nov 2013 17:06:41 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 11522 invoked by uid 99); 26 Nov 2013 17:06:28 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Nov 2013 17:06:28 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=HTML_FONT_FACE_BAD,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of orenault@hortonworks.com designates 209.85.128.182 as permitted sender) Received: from [209.85.128.182] (HELO mail-ve0-f182.google.com) (209.85.128.182) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Nov 2013 17:06:23 +0000 Received: by mail-ve0-f182.google.com with SMTP id jy13so4367299veb.27 for ; Tue, 26 Nov 2013 09:06:02 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=ih6u1mEptp5uabTkqTVL+8wu+y8t9SV/0HgsBk9D1PE=; b=WtPstFLtaC0/6wFHjTH3zRkWFnNFQXC3qw0ROdMIJYJeacoZXA7Lts747oos67ztUu CoDb9R3kDHw3OEFtM5zOmbhnta9yepd39SGq+dX+AUICSw6ZG9QHfzjCL6f+he7I+aKq n8ySVKCixS2yxso4HsFYOmDT4NcQeI4uyyiXQSnhsC22avHnU5ZYoI0ct3DE/8YvFKyh HTFmeo/Gi0a89337zN2ErPk0uYe1xcrTXnkC8JrFhf62lvvm5I7mBUZKMttQ0xgOch9x MEKPjD/4M5oyLXisoiwjaKx/K/ZWzmKfq7YL1aCRPEXvSB1OVsDGNerV5DtEkrAuVcnS 6Pmg== X-Gm-Message-State: ALoCoQlrtuV+ddp5WkwmBry3oGx2Conh7IWrnp/EHZhwKu2gwcvcLZESRqX7g1by38HmCe3BQfnBC452nZoJdO/kbkWdP49BsoCq6pOaJScHtjLVL13f3JU= MIME-Version: 1.0 X-Received: by 10.58.118.84 with SMTP id kk20mr6001203veb.26.1385485562651; Tue, 26 Nov 2013 09:06:02 -0800 (PST) Received: by 10.58.34.135 with HTTP; Tue, 26 Nov 2013 09:06:02 -0800 (PST) In-Reply-To: References: Date: Tue, 26 Nov 2013 17:06:02 +0000 Message-ID: Subject: Re: Working with Capacity Scheduler From: Olivier Renault To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=089e0122a086cdb33104ec177e42 X-Virus-Checked: Checked by ClamAV on apache.org --089e0122a086cdb33104ec177e42 Content-Type: text/plain; charset=US-ASCII Could you confirm that the queue are running by: mapred queue -list Otherwise what error do you get when submitting a job ? Thanks, Olivier On 26 November 2013 15:33, Munna wrote: > Below configured queue's are showing, then why i cannt able to run the job. > > [root@host ~]# hadoop queue -showacls > DEPRECATED: Use of this script to execute mapred command is deprecated. > Instead use the mapred command for it. > > 13/11/26 05:25:43 INFO service.AbstractService: > Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. > 13/11/26 05:25:44 INFO service.AbstractService: > Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. > Queue acls for user : root > > Queue Operations > ===================== > root ADMINISTER_QUEUE,SUBMIT_APPLICATIONS > exploration ADMINISTER_QUEUE,SUBMIT_APPLICATIONS > a ADMINISTER_QUEUE,SUBMIT_APPLICATIONS > b ADMINISTER_QUEUE,SUBMIT_APPLICATIONS > c ADMINISTER_QUEUE,SUBMIT_APPLICATIONS > production ADMINISTER_QUEUE,SUBMIT_APPLICATIONS > > > > On Tue, Nov 26, 2013 at 8:05 PM, Olivier Renault > wrote: > >> Sorry my mistake, it's : >> mapred queue -showacls >> >> Or you can also use hadoop queue -showacls as suggested by Jitendra >> >> Thanks, >> Olviier >> >> >> On 26 November 2013 14:17, Munna wrote: >> >>> Yes! I have executed same command: >>> >>> [user@host ~]$* yarn queue -showacls* >>> >>> Exception in thread "main" java.lang.NoClassDefFoundError: queue >>> Caused by: java.lang.ClassNotFoundException: queue >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >>> Could not find the main class: queue. Program will exit. >>> >>> >>> >>> On Tue, Nov 26, 2013 at 7:25 PM, Olivier Renault < >>> orenault@hortonworks.com> wrote: >>> >>>> it should be : >>>> yarn queue -showacls >>>> >>>> Olivier >>>> >>>> >>>> On 26 November 2013 13:28, Munna wrote: >>>> >>>>> Hi Olivier, >>>>> >>>>> Thank you for your reply. >>>>> >>>>> As you said, i ran those commands and i am getting following error >>>>> message on both the commands. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *[root@host ~]# sudo -u yarn yarn queue -showacls Exception in thread >>>>> "main" java.lang.NoClassDefFoundError: queueCaused by: >>>>> java.lang.ClassNotFoundException: queue at >>>>> java.net.URLClassLoader$1.run(URLClassLoader.java:202) at >>>>> java.security.AccessController.doPrivileged(Native Method) at >>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:190) at >>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:306) at >>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at >>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:247) Could not find the >>>>> main class: queue. Program will exit.* >>>>> >>>>> Regards, >>>>> Munna >>>>> >>>>> >>>>> >>>>> On Tue, Nov 26, 2013 at 4:10 PM, Olivier Renault < >>>>> orenault@hortonworks.com> wrote: >>>>> >>>>>> Could you maybe send us the output of : >>>>>> - yarn queue -showacls >>>>>> - yarn queue -list >>>>>> >>>>>> Thanks, >>>>>> Olivier >>>>>> >>>>>> >>>>>> On 26 November 2013 05:53, Munna wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I working with Capacity Scheduler on YARN and I have configured >>>>>>> different queues. I can able to see all the queues on RM ui. But, when i >>>>>>> start to run MR jobs with configured user names(yarn,mapred), i am unable >>>>>>> to run the Jobs and job are suspended. Again i set default as FIFO working >>>>>>> fine. >>>>>>> >>>>>>> Can you please help me out sort this issue and configured >>>>>>> Configurations are given below. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> * yarn.scheduler.capacity.root.queues >>>>>>> production,exploration >>>>>>> yarn.scheduler.capacity.root.exploration.queues >>>>>>> a,b,c >>>>>>> yarn.scheduler.capacity.root.capacity >>>>>>> 100 >>>>>>> yarn.scheduler.capacity.root.production.capacity >>>>>>> 70 >>>>>>> yarn.scheduler.capacity.root.exploration.capacity >>>>>>> 30 >>>>>>> yarn.scheduler.capacity.root.exploration.a.capacity >>>>>>> 30 >>>>>>> yarn.scheduler.capacity.root.exploration.b.capacity >>>>>>> 30 >>>>>>> yarn.scheduler.capacity.root.exploration.c.capacity >>>>>>> 40 yarn.scheduler.capacity.root. >>>>>>> production.acl_submit_applications >>>>>>> yarn,mapred userb >>>>>>> yarn.scheduler.capacity.root.exploration.a.acl_submit_applications >>>>>>> >>>>>>> yarn.scheduler.capacity.root.exploration.b.acl_submit_applications >>>>>>> userb >>>>>>> yarn.scheduler.capacity.root.exploration.c.acl_submit_applications >>>>>>> userc* >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Regards* >>>>>>> >>>>>>> *Munna* >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. --089e0122a086cdb33104ec177e42 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Could you confirm that the queue are running by:
mapre= d queue -list

Otherwise what error do you get when= submitting a job ?

Thanks,=A0
Olivier


On 26 Novembe= r 2013 15:33, Munna <munnavarsk@gmail.com> wrote:
Below configured queue's are showing, then why i cannt= able to run the job.

[root@host ~]# hadoop queue -showacls
DEPRE= CATED: Use of this script to execute mapred command is deprecated.
Inste= ad use the mapred command for it.

13/11/26 05:25:43 INFO service.AbstractService: Service:org.apache.hado= op.yarn.client.YarnClientImpl is inited.
13/11/26 05:25:44 INFO service.= AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is st= arted.
Queue acls for user :=A0 root

Queue=A0 Operations
=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
root=A0 ADMINISTER_QUEU= E,SUBMIT_APPLICATIONS
exploration=A0 ADMINISTER_QUEUE,SUBMIT_APPLICATION= S
a=A0 ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
b=A0 ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
c=A0 ADMINISTER_QUEUE,SUBMIT_A= PPLICATIONS
production=A0 ADMINISTER_QUEUE,SUBMIT_APPLICATIONS


=
On Tue, Nov 26, 2013 at 8:05 PM, Olivier Renault <orenault@hortonwo= rks.com> wrote:
Sorry my mistake, it's = :=A0
mapred queue -showacls

Or you can also us= e hadoop queue -showacls as suggested by=A0Jitendra

Thanks,=A0<= /div>
Olviier


On 26 November 2013 14:17, Munna <munnavarsk@gmail.com> wrote:
Yes! I have executed same command:
[user@host ~]$ yarn queue -showacls

Exception in thread "main" java.lang.NoClassDefFoundError: qu= eue
Caused by: java.lang.ClassNotFoundException: queue
=A0=A0=A0=A0=A0=A0=A0 at java.net.URLClassLoader$1.run(URLClassLoader.java:= 202)
=A0=A0=A0=A0=A0=A0=A0 at java.security.AccessController.doPrivilege= d(Native Method)
=A0=A0=A0=A0=A0=A0=A0 at java.net.URLClassLoader.findCl= ass(URLClassLoader.java:190)
=A0=A0=A0=A0=A0=A0=A0 at java.lang.ClassLoa= der.loadClass(ClassLoader.java:306)
=A0=A0=A0=A0=A0=A0=A0 at sun.misc.Launcher$AppClassLoader.loadClass(Launche= r.java:301)
=A0=A0=A0=A0=A0=A0=A0 at java.lang.ClassLoader.loadClass(Cla= ssLoader.java:247)
Could not find the main class: queue.=A0 Program will= exit.



On Tue, Nov 26, 2013 at 7:25 PM, Olivier= Renault <orenault@hortonworks.com> wrote:
it should be :
yarn queue -showacls

Olivier
=


On 26 November 2013 13:28, Munna <munn= avarsk@gmail.com> wrote:
Hi Olivier,

Thank you for your reply.

As you said, i ran those commands an= d i am getting following error message on both the commands.

[root@host ~]# sudo -u yarn yarn queue -showacls
Exception in thread "main" java.lang.NoClassDefFoundError: queue<= br>Caused by: java.lang.ClassNotFoundException: queue
=A0=A0=A0=A0=A0=A0= =A0 at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
=A0=A0=A0= =A0=A0=A0=A0 at java.security.AccessController.doPrivileged(Native Method)<= br> =A0=A0=A0=A0=A0=A0=A0 at java.net.URLClassLoader.findClass(URLClassLoader.j= ava:190)
=A0=A0=A0=A0=A0=A0=A0 at java.lang.ClassLoader.loadClass(ClassL= oader.java:306)
=A0=A0=A0=A0=A0=A0=A0 at sun.misc.Launcher$AppClassLoade= r.loadClass(Launcher.java:301)
=A0=A0=A0=A0=A0=A0=A0 at java.lang.ClassL= oader.loadClass(ClassLoader.java:247)
Could not find the main class: queue.=A0 Program will exit.

Regards,
Munna



On Tue, Nov 26, 2013 at 4:10 PM, Olivier Renault <orenault@hortonwo= rks.com> wrote:
Could you maybe send us the o= utput of :
=A0- yarn queue -showacls
=A0- yarn queue -list

Thanks,=A0
Olivier


On 26 November 2013 05:53, Munna <munnavarsk@gmail.com> w= rote:
Hi,

I working with Capacity Schedul= er on YARN and I have configured different queues. I can able to see all th= e queues on RM ui. But, when i start to run MR jobs with configured user na= mes(yarn,mapred), i am unable to run the Jobs and job are suspended. Again = i set default as FIFO working fine.

Can you please help me out sort this issue and configured Configu= rations are given below.

<property>
=A0=A0=A0 <name&g= t;yarn.scheduler.capacity.root.queues</name>
=A0=A0=A0 <value&g= t;production,exploration</value>
</property>
<property>
=A0=A0=A0 <name>yarn.schedul= er.capacity.root.exploration.queues</name>
=A0=A0=A0 <value>= a,b,c</value>
</property>

<property>
=A0=A0= =A0 <name>yarn.scheduler.capacity.root.capacity</name>
=A0=A0=A0 <value>100</value>
</property>

<pr= operty>
=A0=A0=A0 <name>yarn.scheduler.capacity.root.production= .capacity</name>
=A0=A0=A0 <value>70</value>
</p= roperty>
<property>
=A0=A0=A0 <name>yarn.scheduler.capacity.root.expl= oration.capacity</name>
=A0=A0=A0 <value>30</value></property>

<property>
=A0=A0=A0 <name>yarn.sc= heduler.capacity.root.exploration.a.capacity</name>
=A0=A0=A0 <value>30</value>
</property>

<pro= perty>
=A0=A0=A0 <name>yarn.scheduler.capacity.root.exploration= .b.capacity</name>
=A0=A0=A0 <value>30</value>
<= /property>

<property>
=A0=A0=A0 <name>yarn.scheduler.capacity.root.= exploration.c.capacity</name>
=A0=A0=A0 <value>40</value&= gt;
</property>


<property>
=A0 <name>yar= n.scheduler.capacity.root. production.acl_submit_applications</name><= br> =A0 <value>yarn,mapred</value>
</property>

<= property>
=A0=A0=A0 <value>userb</value>
=A0=A0=A0 <= ;name>yarn.scheduler.capacity.root.exploration.a.acl_submit_applications= </name>
</property>

<property>
=A0=A0=A0 <name>yarn.sch= eduler.capacity.root.exploration.b.acl_submit_applications</name>
= =A0=A0=A0 <value>userb</value>
</property>

<= property>
=A0=A0=A0 <name>yarn.scheduler.capacity.root.exploration.c.acl_submit= _applications</name>
=A0=A0=A0 <value>userc</value></property>

</configuration>


--
Regards

Munna



<= /blockquote>
<= /div>

CONFIDENTIALITY NOTICE
NOTICE: This message is = intended for the use of the individual or entity to which it is addressed a= nd may contain information that is confidential, privileged and exempt from= disclosure under applicable law. If the reader of this message is not the = intended recipient, you are hereby notified that any printing, copying, dis= semination, distribution, disclosure or forwarding of this communication is= strictly prohibited. If you have received this communication in error, ple= ase contact the sender immediately and delete it from your system. Thank Yo= u. --089e0122a086cdb33104ec177e42--