hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Munna <munnava...@gmail.com>
Subject Re: Working with Capacity Scheduler
Date Tue, 26 Nov 2013 17:47:56 GMT
*mapred queue -list* result:

[user@host ~]$ mapred queue -list
13/11/26 09:37:48 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
13/11/26 09:37:48 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
======================
Queue Name : exploration
Queue State : running
Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0,
CurrentCapacity: 0.0
    ======================
    Queue Name : a
    Queue State : running
    Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0,
CurrentCapacity: 0.0
    ======================
    Queue Name : b
    Queue State : running
    Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0,
CurrentCapacity: 0.0
    ======================
    Queue Name : c
    Queue State : running
    Scheduling Info : Capacity: 40.0, MaximumCapacity: 1.0,
CurrentCapacity: 0.0
======================
Queue Name : production
Queue State : running
Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, CurrentCapacity: 0.0

and I saw in RM logs, mapred user try to find default queue, logs are below:

2013-11-26 05:22:59,804 INFO
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application
with id 1 submitted by user mapred
2013-11-26 05:22:59,808 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=mapred
IP=10.10.10.1  OPERATION=Submit Application Request
TARGET=ClientRMService  RESULT=SUCCESS  APPID=application_1385472086605_0001
2013-11-26 05:22:59,828 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
application_1385472086605_0001 State change from NEW to SUBMITTED
2013-11-26 05:22:59,829 INFO
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
Registering appattempt_1385472086605_0001_000001
2013-11-26 05:22:59,830 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1385472086605_0001_000001 State change from NEW to SUBMITTED
2013-11-26 05:22:59,832 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1385472086605_0001_000001 State change from SUBMITTED to FAILED
2013-11-26 05:22:59,836 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
application_1385472086605_0001 State change from SUBMITTED to FAILED
2013-11-26 05:22:59,837 WARN
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=mapred
OPERATION=Application Finished - Failed TARGET=RMAppManager
RESULT=FAILURE  DESCRIPTION=App failed with state: FAILED
PERMISSIONS=Application appattempt_1385472086605_0001_000001 submitted by
user mapred to unknown queue: default APPID=application_1385472086605_0001
2013-11-26 05:22:59,841 INFO
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore:
Removing info for app: application_1385472086605_0001
2013-11-26 05:22:59,842 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary:
appId=application_1385472086605_0001,name=select count(*) from
crc_exchange_rate(Stage-1),user=mapred,queue=default,state=FAILED,trackingUrl=hostname:8088/proxy/application_1385472086605_0001/,appMasterHost=N/A,startTime=1385472179799,finishTime=1385472179836



On Tue, Nov 26, 2013 at 10:36 PM, Olivier Renault
<orenault@hortonworks.com>wrote:

> Could you confirm that the queue are running by:
> mapred queue -list
>
> Otherwise what error do you get when submitting a job ?
>
> Thanks,
> Olivier
>
>
> On 26 November 2013 15:33, Munna <munnavarsk@gmail.com> wrote:
>
>> Below configured queue's are showing, then why i cannt able to run the
>> job.
>>
>> [root@host ~]# hadoop queue -showacls
>> DEPRECATED: Use of this script to execute mapred command is deprecated.
>> Instead use the mapred command for it.
>>
>> 13/11/26 05:25:43 INFO service.AbstractService:
>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
>> 13/11/26 05:25:44 INFO service.AbstractService:
>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
>> Queue acls for user :  root
>>
>> Queue  Operations
>> =====================
>> root  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>> exploration  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>> a  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>> b  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>> c  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>> production  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>>
>>
>>
>> On Tue, Nov 26, 2013 at 8:05 PM, Olivier Renault <
>> orenault@hortonworks.com> wrote:
>>
>>> Sorry my mistake, it's :
>>> mapred queue -showacls
>>>
>>> Or you can also use hadoop queue -showacls as suggested by Jitendra
>>>
>>> Thanks,
>>> Olviier
>>>
>>>
>>> On 26 November 2013 14:17, Munna <munnavarsk@gmail.com> wrote:
>>>
>>>> Yes! I have executed same command:
>>>>
>>>> [user@host ~]$* yarn queue -showacls*
>>>>
>>>> Exception in thread "main" java.lang.NoClassDefFoundError: queue
>>>> Caused by: java.lang.ClassNotFoundException: queue
>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>> Could not find the main class: queue.  Program will exit.
>>>>
>>>>
>>>>
>>>> On Tue, Nov 26, 2013 at 7:25 PM, Olivier Renault <
>>>> orenault@hortonworks.com> wrote:
>>>>
>>>>> it should be :
>>>>> yarn queue -showacls
>>>>>
>>>>> Olivier
>>>>>
>>>>>
>>>>> On 26 November 2013 13:28, Munna <munnavarsk@gmail.com> wrote:
>>>>>
>>>>>> Hi Olivier,
>>>>>>
>>>>>> Thank you for your reply.
>>>>>>
>>>>>> As you said, i ran those commands and i am getting following error
>>>>>> message on both the commands.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *[root@host ~]# sudo -u yarn yarn queue -showacls Exception in thread
>>>>>> "main" java.lang.NoClassDefFoundError: queueCaused by:
>>>>>> java.lang.ClassNotFoundException: queue        at
>>>>>> java.net.URLClassLoader$1.run(URLClassLoader.java:202)        at
>>>>>> java.security.AccessController.doPrivileged(Native Method)      
  at
>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:190)      
 at
>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:306)        at
>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)   
    at
>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:247) Could not find
the
>>>>>> main class: queue.  Program will exit.*
>>>>>>
>>>>>> Regards,
>>>>>> Munna
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Nov 26, 2013 at 4:10 PM, Olivier Renault <
>>>>>> orenault@hortonworks.com> wrote:
>>>>>>
>>>>>>> Could you maybe send us the output of :
>>>>>>>  - yarn queue -showacls
>>>>>>>  - yarn queue -list
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Olivier
>>>>>>>
>>>>>>>
>>>>>>> On 26 November 2013 05:53, Munna <munnavarsk@gmail.com>
wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I working with Capacity Scheduler on YARN and I have configured
>>>>>>>> different queues. I can able to see all the queues on RM
ui. But, when i
>>>>>>>> start to run MR jobs with configured user names(yarn,mapred),
i am unable
>>>>>>>> to run the Jobs and job are suspended. Again i set default
as FIFO working
>>>>>>>> fine.
>>>>>>>>
>>>>>>>> Can you please help me out sort this issue and configured
>>>>>>>> Configurations are given below.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *<property>    <name>yarn.scheduler.capacity.root.queues</name>
>>>>>>>> <value>production,exploration</value> </property><property>
>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.queues</name>
>>>>>>>> <value>a,b,c</value></property><property>
>>>>>>>> <name>yarn.scheduler.capacity.root.capacity</name>
>>>>>>>> <value>100</value></property><property>
>>>>>>>> <name>yarn.scheduler.capacity.root.production.capacity</name>
>>>>>>>> <value>70</value></property> <property>
>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.capacity</name>
>>>>>>>> <value>30</value></property><property>
>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.a.capacity</name>
>>>>>>>> <value>30</value></property><property>
>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.b.capacity</name>
>>>>>>>> <value>30</value></property> <property>
>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.c.capacity</name>
>>>>>>>> <value>40</value></property><property>
 <name>yarn.scheduler.capacity.root.
>>>>>>>> production.acl_submit_applications</name>
>>>>>>>> <value>yarn,mapred</value></property><property>
   <value>userb</value>
>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.a.acl_submit_applications</name>
>>>>>>>> </property><property>
>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.b.acl_submit_applications</name>
>>>>>>>> <value>userb</value></property><property>
>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.c.acl_submit_applications</name>
>>>>>>>> <value>userc</value></property></configuration>*
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Regards*
>>>>>>>>
>>>>>>>>  *Munna*
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 
*Regards*

*Munna*

Mime
View raw message