hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Munna <munnava...@gmail.com>
Subject Re: Working with Capacity Scheduler
Date Tue, 26 Nov 2013 20:14:16 GMT
I am using Hadoop 2.0.0 and YARN 2.0.0,... is it require to set any
parameter on mapred-site.xml or yarn-site.xml?


On Wed, Nov 27, 2013 at 1:03 AM, Olivier Renault
<orenault@hortonworks.com>wrote:

> When launching a job, you need to specify the queue.
> "...
> mapred to unknown queue
> ... "
>
> Something along the line of (this is for hadoop1)
>
> -Dmapred.queue.name=<queue name>
>
> Thanks
> Olivier
> *mapred queue -list* result:
>
> [user@host ~]$ mapred queue -list
> 13/11/26 09:37:48 INFO service.AbstractService:
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
> 13/11/26 09:37:48 INFO service.AbstractService:
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
> ======================
> Queue Name : exploration
> Queue State : running
> Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0,
> CurrentCapacity: 0.0
>     ======================
>     Queue Name : a
>     Queue State : running
>     Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0,
> CurrentCapacity: 0.0
>     ======================
>     Queue Name : b
>     Queue State : running
>     Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0,
> CurrentCapacity: 0.0
>     ======================
>     Queue Name : c
>     Queue State : running
>     Scheduling Info : Capacity: 40.0, MaximumCapacity: 1.0,
> CurrentCapacity: 0.0
> ======================
> Queue Name : production
> Queue State : running
> Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, CurrentCapacity:
> 0.0
>
> and I saw in RM logs, mapred user try to find default queue, logs are
> below:
>
> 2013-11-26 05:22:59,804 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application
> with id 1 submitted by user mapred
> 2013-11-26 05:22:59,808 INFO
> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=mapred
> IP=10.10.10.1  OPERATION=Submit Application Request
> TARGET=ClientRMService  RESULT=SUCCESS  APPID=application_1385472086605_0001
> 2013-11-26 05:22:59,828 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
> application_1385472086605_0001 State change from NEW to SUBMITTED
> 2013-11-26 05:22:59,829 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
> Registering appattempt_1385472086605_0001_000001
> 2013-11-26 05:22:59,830 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
> appattempt_1385472086605_0001_000001 State change from NEW to SUBMITTED
> 2013-11-26 05:22:59,832 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
> appattempt_1385472086605_0001_000001 State change from SUBMITTED to FAILED
> 2013-11-26 05:22:59,836 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
> application_1385472086605_0001 State change from SUBMITTED to FAILED
> 2013-11-26 05:22:59,837 WARN
> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=mapred
> OPERATION=Application Finished - Failed TARGET=RMAppManager
> RESULT=FAILURE  DESCRIPTION=App failed with state: FAILED
> PERMISSIONS=Application appattempt_1385472086605_0001_000001 submitted by
> user mapred to unknown queue: default APPID=application_1385472086605_0001
> 2013-11-26 05:22:59,841 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore:
> Removing info for app: application_1385472086605_0001
> 2013-11-26 05:22:59,842 INFO
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary:
> appId=application_1385472086605_0001,name=select count(*) from
> crc_exchange_rate(Stage-1),user=mapred,queue=default,state=FAILED,trackingUrl=hostname:8088/proxy/application_1385472086605_0001/,appMasterHost=N/A,startTime=1385472179799,finishTime=1385472179836
>
>
>
> On Tue, Nov 26, 2013 at 10:36 PM, Olivier Renault <
> orenault@hortonworks.com> wrote:
>
>> Could you confirm that the queue are running by:
>> mapred queue -list
>>
>> Otherwise what error do you get when submitting a job ?
>>
>> Thanks,
>> Olivier
>>
>>
>> On 26 November 2013 15:33, Munna <munnavarsk@gmail.com> wrote:
>>
>>> Below configured queue's are showing, then why i cannt able to run the
>>> job.
>>>
>>> [root@host ~]# hadoop queue -showacls
>>> DEPRECATED: Use of this script to execute mapred command is deprecated.
>>> Instead use the mapred command for it.
>>>
>>> 13/11/26 05:25:43 INFO service.AbstractService:
>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
>>> 13/11/26 05:25:44 INFO service.AbstractService:
>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
>>> Queue acls for user :  root
>>>
>>> Queue  Operations
>>> =====================
>>> root  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>>> exploration  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>>> a  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>>> b  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>>> c  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>>> production  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>>>
>>>
>>>
>>> On Tue, Nov 26, 2013 at 8:05 PM, Olivier Renault <
>>> orenault@hortonworks.com> wrote:
>>>
>>>> Sorry my mistake, it's :
>>>> mapred queue -showacls
>>>>
>>>> Or you can also use hadoop queue -showacls as suggested by Jitendra
>>>>
>>>> Thanks,
>>>> Olviier
>>>>
>>>>
>>>> On 26 November 2013 14:17, Munna <munnavarsk@gmail.com> wrote:
>>>>
>>>>> Yes! I have executed same command:
>>>>>
>>>>> [user@host ~]$* yarn queue -showacls*
>>>>>
>>>>> Exception in thread "main" java.lang.NoClassDefFoundError: queue
>>>>> Caused by: java.lang.ClassNotFoundException: queue
>>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>>         at
>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>> Could not find the main class: queue.  Program will exit.
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Nov 26, 2013 at 7:25 PM, Olivier Renault <
>>>>> orenault@hortonworks.com> wrote:
>>>>>
>>>>>> it should be :
>>>>>> yarn queue -showacls
>>>>>>
>>>>>> Olivier
>>>>>>
>>>>>>
>>>>>> On 26 November 2013 13:28, Munna <munnavarsk@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Olivier,
>>>>>>>
>>>>>>> Thank you for your reply.
>>>>>>>
>>>>>>> As you said, i ran those commands and i am getting following
error
>>>>>>> message on both the commands.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *[root@host ~]# sudo -u yarn yarn queue -showacls Exception in
>>>>>>> thread "main" java.lang.NoClassDefFoundError: queueCaused by:
>>>>>>> java.lang.ClassNotFoundException: queue        at
>>>>>>> java.net.URLClassLoader$1.run(URLClassLoader.java:202)      
 at
>>>>>>> java.security.AccessController.doPrivileged(Native Method)  
      at
>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:190)  
     at
>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:306)       
at
>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
       at
>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:247) Could not
find the
>>>>>>> main class: queue.  Program will exit.*
>>>>>>>
>>>>>>> Regards,
>>>>>>> Munna
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Nov 26, 2013 at 4:10 PM, Olivier Renault <
>>>>>>> orenault@hortonworks.com> wrote:
>>>>>>>
>>>>>>>> Could you maybe send us the output of :
>>>>>>>>  - yarn queue -showacls
>>>>>>>>  - yarn queue -list
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Olivier
>>>>>>>>
>>>>>>>>
>>>>>>>> On 26 November 2013 05:53, Munna <munnavarsk@gmail.com>
wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I working with Capacity Scheduler on YARN and I have
configured
>>>>>>>>> different queues. I can able to see all the queues on
RM ui. But, when i
>>>>>>>>> start to run MR jobs with configured user names(yarn,mapred),
i am unable
>>>>>>>>> to run the Jobs and job are suspended. Again i set default
as FIFO working
>>>>>>>>> fine.
>>>>>>>>>
>>>>>>>>> Can you please help me out sort this issue and configured
>>>>>>>>> Configurations are given below.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *<property>    <name>yarn.scheduler.capacity.root.queues</name>
>>>>>>>>> <value>production,exploration</value> </property><property>
>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.queues</name>
>>>>>>>>> <value>a,b,c</value></property><property>
>>>>>>>>> <name>yarn.scheduler.capacity.root.capacity</name>
>>>>>>>>> <value>100</value></property><property>
>>>>>>>>> <name>yarn.scheduler.capacity.root.production.capacity</name>
>>>>>>>>> <value>70</value></property> <property>
>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.capacity</name>
>>>>>>>>> <value>30</value></property><property>
>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.a.capacity</name>
>>>>>>>>> <value>30</value></property><property>
>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.b.capacity</name>
>>>>>>>>> <value>30</value></property> <property>
>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.c.capacity</name>
>>>>>>>>> <value>40</value></property><property>
 <name>yarn.scheduler.capacity.root.
>>>>>>>>> production.acl_submit_applications</name>
>>>>>>>>> <value>yarn,mapred</value></property><property>
   <value>userb</value>
>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.a.acl_submit_applications</name>
>>>>>>>>> </property><property>
>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.b.acl_submit_applications</name>
>>>>>>>>> <value>userb</value></property><property>
>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.c.acl_submit_applications</name>
>>>>>>>>> <value>userc</value></property></configuration>*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Regards*
>>>>>>>>>
>>>>>>>>>  *Munna*
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
>
> --
> *Regards*
>
> *Munna*
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 
*Regards*

*Munna*

Mime
View raw message