hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Munna <munnava...@gmail.com>
Subject Re: Working with Capacity Scheduler
Date Tue, 26 Nov 2013 21:48:38 GMT
Hi,

Can i able to get the solution on Capacity Scheduler ?


On Wed, Nov 27, 2013 at 2:24 AM, Munna <munnavarsk@gmail.com> wrote:

> yes... acl's are not enforced.
>
>
> On Wed, Nov 27, 2013 at 2:16 AM, Olivier Renault <orenault@hortonworks.com
> > wrote:
>
>> Do you mean that acl are not being enforced?
>>
>> Olivier
>> On 26 Nov 2013 20:18, "Munna" <munnavarsk@gmail.com> wrote:
>>
>>> Hi Olivier,
>>>
>>> have u hold any solution, why user are not identifying their queues?
>>>
>>> Thanks
>>> Munna
>>>
>>>
>>> On Tue, Nov 26, 2013 at 11:17 PM, Munna <munnavarsk@gmail.com> wrote:
>>>
>>>> *mapred queue -list* result:
>>>>
>>>> [user@host ~]$ mapred queue -list
>>>> 13/11/26 09:37:48 INFO service.AbstractService:
>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
>>>> 13/11/26 09:37:48 INFO service.AbstractService:
>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
>>>> ======================
>>>> Queue Name : exploration
>>>> Queue State : running
>>>> Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0,
>>>> CurrentCapacity: 0.0
>>>>     ======================
>>>>     Queue Name : a
>>>>     Queue State : running
>>>>     Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0,
>>>> CurrentCapacity: 0.0
>>>>     ======================
>>>>     Queue Name : b
>>>>     Queue State : running
>>>>     Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0,
>>>> CurrentCapacity: 0.0
>>>>     ======================
>>>>     Queue Name : c
>>>>     Queue State : running
>>>>     Scheduling Info : Capacity: 40.0, MaximumCapacity: 1.0,
>>>> CurrentCapacity: 0.0
>>>> ======================
>>>> Queue Name : production
>>>> Queue State : running
>>>> Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0,
>>>> CurrentCapacity: 0.0
>>>>
>>>> and I saw in RM logs, mapred user try to find default queue, logs are
>>>> below:
>>>>
>>>> 2013-11-26 05:22:59,804 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application
>>>> with id 1 submitted by user mapred
>>>> 2013-11-26 05:22:59,808 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=mapred
>>>> IP=10.10.10.1  OPERATION=Submit Application Request
>>>> TARGET=ClientRMService  RESULT=SUCCESS  APPID=application_1385472086605_0001
>>>> 2013-11-26 05:22:59,828 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
>>>> application_1385472086605_0001 State change from NEW to SUBMITTED
>>>> 2013-11-26 05:22:59,829 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
>>>> Registering appattempt_1385472086605_0001_000001
>>>> 2013-11-26 05:22:59,830 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
>>>> appattempt_1385472086605_0001_000001 State change from NEW to SUBMITTED
>>>> 2013-11-26 05:22:59,832 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
>>>> appattempt_1385472086605_0001_000001 State change from SUBMITTED to FAILED
>>>> 2013-11-26 05:22:59,836 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
>>>> application_1385472086605_0001 State change from SUBMITTED to FAILED
>>>> 2013-11-26 05:22:59,837 WARN
>>>> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=mapred
>>>> OPERATION=Application Finished - Failed TARGET=RMAppManager
>>>> RESULT=FAILURE  DESCRIPTION=App failed with state: FAILED
>>>> PERMISSIONS=Application appattempt_1385472086605_0001_000001 submitted by
>>>> user mapred to unknown queue: default APPID=application_1385472086605_0001
>>>> 2013-11-26 05:22:59,841 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore:
>>>> Removing info for app: application_1385472086605_0001
>>>> 2013-11-26 05:22:59,842 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary:
>>>> appId=application_1385472086605_0001,name=select count(*) from
>>>> crc_exchange_rate(Stage-1),user=mapred,queue=default,state=FAILED,trackingUrl=hostname:8088/proxy/application_1385472086605_0001/,appMasterHost=N/A,startTime=1385472179799,finishTime=1385472179836
>>>>
>>>>
>>>>
>>>> On Tue, Nov 26, 2013 at 10:36 PM, Olivier Renault <
>>>> orenault@hortonworks.com> wrote:
>>>>
>>>>> Could you confirm that the queue are running by:
>>>>> mapred queue -list
>>>>>
>>>>> Otherwise what error do you get when submitting a job ?
>>>>>
>>>>> Thanks,
>>>>> Olivier
>>>>>
>>>>>
>>>>> On 26 November 2013 15:33, Munna <munnavarsk@gmail.com> wrote:
>>>>>
>>>>>> Below configured queue's are showing, then why i cannt able to run
>>>>>> the job.
>>>>>>
>>>>>> [root@host ~]# hadoop queue -showacls
>>>>>> DEPRECATED: Use of this script to execute mapred command is
>>>>>> deprecated.
>>>>>> Instead use the mapred command for it.
>>>>>>
>>>>>> 13/11/26 05:25:43 INFO service.AbstractService:
>>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
>>>>>> 13/11/26 05:25:44 INFO service.AbstractService:
>>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
>>>>>> Queue acls for user :  root
>>>>>>
>>>>>> Queue  Operations
>>>>>> =====================
>>>>>> root  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>>>>>> exploration  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>>>>>> a  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>>>>>> b  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>>>>>> c  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>>>>>> production  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Nov 26, 2013 at 8:05 PM, Olivier Renault <
>>>>>> orenault@hortonworks.com> wrote:
>>>>>>
>>>>>>> Sorry my mistake, it's :
>>>>>>> mapred queue -showacls
>>>>>>>
>>>>>>> Or you can also use hadoop queue -showacls as suggested by Jitendra
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Olviier
>>>>>>>
>>>>>>>
>>>>>>> On 26 November 2013 14:17, Munna <munnavarsk@gmail.com>
wrote:
>>>>>>>
>>>>>>>> Yes! I have executed same command:
>>>>>>>>
>>>>>>>> [user@host ~]$* yarn queue -showacls*
>>>>>>>>
>>>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
queue
>>>>>>>> Caused by: java.lang.ClassNotFoundException: queue
>>>>>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>>>>>         at java.security.AccessController.doPrivileged(Native
>>>>>>>> Method)
>>>>>>>>         at
>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>>>>>         at
>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>>>>> Could not find the main class: queue.  Program will exit.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Nov 26, 2013 at 7:25 PM, Olivier Renault <
>>>>>>>> orenault@hortonworks.com> wrote:
>>>>>>>>
>>>>>>>>> it should be :
>>>>>>>>> yarn queue -showacls
>>>>>>>>>
>>>>>>>>> Olivier
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 26 November 2013 13:28, Munna <munnavarsk@gmail.com>
wrote:
>>>>>>>>>
>>>>>>>>>> Hi Olivier,
>>>>>>>>>>
>>>>>>>>>> Thank you for your reply.
>>>>>>>>>>
>>>>>>>>>> As you said, i ran those commands and i am getting
following
>>>>>>>>>> error message on both the commands.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *[root@host ~]# sudo -u yarn yarn queue -showacls
Exception in
>>>>>>>>>> thread "main" java.lang.NoClassDefFoundError: queueCaused
by:
>>>>>>>>>> java.lang.ClassNotFoundException: queue        at
>>>>>>>>>> java.net.URLClassLoader$1.run(URLClassLoader.java:202)
       at
>>>>>>>>>> java.security.AccessController.doPrivileged(Native
Method)         at
>>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:190)
       at
>>>>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:306)
       at
>>>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
       at
>>>>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the
>>>>>>>>>> main class: queue.  Program will exit.*
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Munna
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Nov 26, 2013 at 4:10 PM, Olivier Renault
<
>>>>>>>>>> orenault@hortonworks.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Could you maybe send us the output of :
>>>>>>>>>>>  - yarn queue -showacls
>>>>>>>>>>>  - yarn queue -list
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Olivier
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 26 November 2013 05:53, Munna <munnavarsk@gmail.com>
wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I working with Capacity Scheduler on YARN
and I have configured
>>>>>>>>>>>> different queues. I can able to see all the
queues on RM ui. But, when i
>>>>>>>>>>>> start to run MR jobs with configured user
names(yarn,mapred), i am unable
>>>>>>>>>>>> to run the Jobs and job are suspended. Again
i set default as FIFO working
>>>>>>>>>>>> fine.
>>>>>>>>>>>>
>>>>>>>>>>>> Can you please help me out sort this issue
and configured
>>>>>>>>>>>> Configurations are given below.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> *<property>
>>>>>>>>>>>> <name>yarn.scheduler.capacity.root.queues</name>
>>>>>>>>>>>> <value>production,exploration</value>
</property><property>
>>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.queues</name>
>>>>>>>>>>>> <value>a,b,c</value></property><property>
>>>>>>>>>>>> <name>yarn.scheduler.capacity.root.capacity</name>
>>>>>>>>>>>> <value>100</value></property><property>
>>>>>>>>>>>> <name>yarn.scheduler.capacity.root.production.capacity</name>
>>>>>>>>>>>> <value>70</value></property>
<property>
>>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.capacity</name>
>>>>>>>>>>>> <value>30</value></property><property>
>>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.a.capacity</name>
>>>>>>>>>>>> <value>30</value></property><property>
>>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.b.capacity</name>
>>>>>>>>>>>> <value>30</value></property>
<property>
>>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.c.capacity</name>
>>>>>>>>>>>> <value>40</value></property><property>
 <name>yarn.scheduler.capacity.root.
>>>>>>>>>>>> production.acl_submit_applications</name>
>>>>>>>>>>>> <value>yarn,mapred</value></property><property>
   <value>userb</value>
>>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.a.acl_submit_applications</name>
>>>>>>>>>>>> </property><property>
>>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.b.acl_submit_applications</name>
>>>>>>>>>>>> <value>userb</value></property><property>
>>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.c.acl_submit_applications</name>
>>>>>>>>>>>> <value>userc</value></property></configuration>*
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> *Regards*
>>>>>>>>>>>>
>>>>>>>>>>>>  *Munna*
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity to which it is addressed and may contain information that is
>>>>> confidential, privileged and exempt from disclosure under applicable
law.
>>>>> If the reader of this message is not the intended recipient, you are
hereby
>>>>> notified that any printing, copying, dissemination, distribution,
>>>>> disclosure or forwarding of this communication is strictly prohibited.
If
>>>>> you have received this communication in error, please contact the sender
>>>>> immediately and delete it from your system. Thank You.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Regards*
>>>>
>>>> *Munna*
>>>>
>>>
>>>
>>>
>>> --
>>> *Regards*
>>>
>>> *Munna*
>>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
>
> --
> *Regards*
>
> *Munna*
>



-- 
*Regards*

*Munna*

Mime
View raw message