hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Munna <munnava...@gmail.com>
Subject Re: Working with Capacity Scheduler
Date Tue, 26 Nov 2013 22:51:43 GMT
Hi Olivier,

"yarn.acl.enable"  is enabled earlier, for your information i am using
Cloudera Manager to manage the cluster. again same problem :(




On Wed, Nov 27, 2013 at 3:37 AM, Olivier Renault
<orenault@hortonworks.com>wrote:

> Here is a working configuration.
>
> In yarn-site.xml, you'll need to enable
>     <name>yarn.acl.enable</name>
>     <value>true</value>
>
> Thanks,
> Olivier
>
>
> On 26 November 2013 21:48, Munna <munnavarsk@gmail.com> wrote:
>
>> Hi,
>>
>> Can i able to get the solution on Capacity Scheduler ?
>>
>>
>> On Wed, Nov 27, 2013 at 2:24 AM, Munna <munnavarsk@gmail.com> wrote:
>>
>>> yes... acl's are not enforced.
>>>
>>>
>>> On Wed, Nov 27, 2013 at 2:16 AM, Olivier Renault <
>>> orenault@hortonworks.com> wrote:
>>>
>>>> Do you mean that acl are not being enforced?
>>>>
>>>> Olivier
>>>> On 26 Nov 2013 20:18, "Munna" <munnavarsk@gmail.com> wrote:
>>>>
>>>>> Hi Olivier,
>>>>>
>>>>> have u hold any solution, why user are not identifying their queues?
>>>>>
>>>>> Thanks
>>>>> Munna
>>>>>
>>>>>
>>>>> On Tue, Nov 26, 2013 at 11:17 PM, Munna <munnavarsk@gmail.com>
wrote:
>>>>>
>>>>>> *mapred queue -list* result:
>>>>>>
>>>>>> [user@host ~]$ mapred queue -list
>>>>>> 13/11/26 09:37:48 INFO service.AbstractService:
>>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
>>>>>> 13/11/26 09:37:48 INFO service.AbstractService:
>>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
>>>>>> ======================
>>>>>> Queue Name : exploration
>>>>>> Queue State : running
>>>>>> Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0,
>>>>>> CurrentCapacity: 0.0
>>>>>>     ======================
>>>>>>     Queue Name : a
>>>>>>     Queue State : running
>>>>>>     Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0,
>>>>>> CurrentCapacity: 0.0
>>>>>>     ======================
>>>>>>     Queue Name : b
>>>>>>     Queue State : running
>>>>>>     Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0,
>>>>>> CurrentCapacity: 0.0
>>>>>>     ======================
>>>>>>     Queue Name : c
>>>>>>     Queue State : running
>>>>>>     Scheduling Info : Capacity: 40.0, MaximumCapacity: 1.0,
>>>>>> CurrentCapacity: 0.0
>>>>>> ======================
>>>>>> Queue Name : production
>>>>>> Queue State : running
>>>>>> Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0,
>>>>>> CurrentCapacity: 0.0
>>>>>>
>>>>>> and I saw in RM logs, mapred user try to find default queue, logs
are
>>>>>> below:
>>>>>>
>>>>>> 2013-11-26 05:22:59,804 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application
>>>>>> with id 1 submitted by user mapred
>>>>>> 2013-11-26 05:22:59,808 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=mapred
>>>>>> IP=10.10.10.1  OPERATION=Submit Application Request
>>>>>> TARGET=ClientRMService  RESULT=SUCCESS  APPID=application_1385472086605_0001
>>>>>> 2013-11-26 05:22:59,828 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
>>>>>> application_1385472086605_0001 State change from NEW to SUBMITTED
>>>>>> 2013-11-26 05:22:59,829 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
>>>>>> Registering appattempt_1385472086605_0001_000001
>>>>>> 2013-11-26 05:22:59,830 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
>>>>>> appattempt_1385472086605_0001_000001 State change from NEW to SUBMITTED
>>>>>> 2013-11-26 05:22:59,832 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
>>>>>> appattempt_1385472086605_0001_000001 State change from SUBMITTED
to FAILED
>>>>>> 2013-11-26 05:22:59,836 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
>>>>>> application_1385472086605_0001 State change from SUBMITTED to FAILED
>>>>>> 2013-11-26 05:22:59,837 WARN
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=mapred
>>>>>> OPERATION=Application Finished - Failed TARGET=RMAppManager
>>>>>> RESULT=FAILURE  DESCRIPTION=App failed with state: FAILED
>>>>>> PERMISSIONS=Application appattempt_1385472086605_0001_000001 submitted
by
>>>>>> user mapred to unknown queue: default APPID=application_1385472086605_0001
>>>>>> 2013-11-26 05:22:59,841 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore:
>>>>>> Removing info for app: application_1385472086605_0001
>>>>>> 2013-11-26 05:22:59,842 INFO
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary:
>>>>>> appId=application_1385472086605_0001,name=select count(*) from
>>>>>> crc_exchange_rate(Stage-1),user=mapred,queue=default,state=FAILED,trackingUrl=hostname:8088/proxy/application_1385472086605_0001/,appMasterHost=N/A,startTime=1385472179799,finishTime=1385472179836
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Nov 26, 2013 at 10:36 PM, Olivier Renault <
>>>>>> orenault@hortonworks.com> wrote:
>>>>>>
>>>>>>> Could you confirm that the queue are running by:
>>>>>>> mapred queue -list
>>>>>>>
>>>>>>> Otherwise what error do you get when submitting a job ?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Olivier
>>>>>>>
>>>>>>>
>>>>>>> On 26 November 2013 15:33, Munna <munnavarsk@gmail.com>
wrote:
>>>>>>>
>>>>>>>> Below configured queue's are showing, then why i cannt able
to run
>>>>>>>> the job.
>>>>>>>>
>>>>>>>> [root@host ~]# hadoop queue -showacls
>>>>>>>> DEPRECATED: Use of this script to execute mapred command
is
>>>>>>>> deprecated.
>>>>>>>> Instead use the mapred command for it.
>>>>>>>>
>>>>>>>> 13/11/26 05:25:43 INFO service.AbstractService:
>>>>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
>>>>>>>> 13/11/26 05:25:44 INFO service.AbstractService:
>>>>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
>>>>>>>> Queue acls for user :  root
>>>>>>>>
>>>>>>>> Queue  Operations
>>>>>>>> =====================
>>>>>>>> root  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>>>>>>>> exploration  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>>>>>>>> a  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>>>>>>>> b  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>>>>>>>> c  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>>>>>>>> production  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Nov 26, 2013 at 8:05 PM, Olivier Renault <
>>>>>>>> orenault@hortonworks.com> wrote:
>>>>>>>>
>>>>>>>>> Sorry my mistake, it's :
>>>>>>>>> mapred queue -showacls
>>>>>>>>>
>>>>>>>>> Or you can also use hadoop queue -showacls as suggested
by
>>>>>>>>> Jitendra
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Olviier
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 26 November 2013 14:17, Munna <munnavarsk@gmail.com>
wrote:
>>>>>>>>>
>>>>>>>>>> Yes! I have executed same command:
>>>>>>>>>>
>>>>>>>>>> [user@host ~]$* yarn queue -showacls*
>>>>>>>>>>
>>>>>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
queue
>>>>>>>>>> Caused by: java.lang.ClassNotFoundException: queue
>>>>>>>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>>>>>>>         at java.security.AccessController.doPrivileged(Native
>>>>>>>>>> Method)
>>>>>>>>>>         at
>>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>>>>>>>         at
>>>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>>>>>>> Could not find the main class: queue.  Program will
exit.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Nov 26, 2013 at 7:25 PM, Olivier Renault
<
>>>>>>>>>> orenault@hortonworks.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> it should be :
>>>>>>>>>>> yarn queue -showacls
>>>>>>>>>>>
>>>>>>>>>>> Olivier
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 26 November 2013 13:28, Munna <munnavarsk@gmail.com>
wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Olivier,
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you for your reply.
>>>>>>>>>>>>
>>>>>>>>>>>> As you said, i ran those commands and i am
getting following
>>>>>>>>>>>> error message on both the commands.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> *[root@host ~]# sudo -u yarn yarn queue -showacls
Exception in
>>>>>>>>>>>> thread "main" java.lang.NoClassDefFoundError:
queueCaused by:
>>>>>>>>>>>> java.lang.ClassNotFoundException: queue 
      at
>>>>>>>>>>>> java.net.URLClassLoader$1.run(URLClassLoader.java:202)
       at
>>>>>>>>>>>> java.security.AccessController.doPrivileged(Native
Method)         at
>>>>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:190)
       at
>>>>>>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:306)
       at
>>>>>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
       at
>>>>>>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the
>>>>>>>>>>>> main class: queue.  Program will exit.*
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Munna
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Nov 26, 2013 at 4:10 PM, Olivier
Renault <
>>>>>>>>>>>> orenault@hortonworks.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Could you maybe send us the output of
:
>>>>>>>>>>>>>  - yarn queue -showacls
>>>>>>>>>>>>>  - yarn queue -list
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Olivier
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 26 November 2013 05:53, Munna <munnavarsk@gmail.com>
wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I working with Capacity Scheduler
on YARN and I have
>>>>>>>>>>>>>> configured different queues. I can
able to see all the queues on RM ui.
>>>>>>>>>>>>>> But, when i start to run MR jobs
with configured user names(yarn,mapred), i
>>>>>>>>>>>>>> am unable to run the Jobs and job
are suspended. Again i set default as
>>>>>>>>>>>>>> FIFO working fine.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can you please help me out sort this
issue and configured
>>>>>>>>>>>>>> Configurations are given below.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *<property>
>>>>>>>>>>>>>> <name>yarn.scheduler.capacity.root.queues</name>
>>>>>>>>>>>>>> <value>production,exploration</value>
</property><property>
>>>>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.queues</name>
>>>>>>>>>>>>>> <value>a,b,c</value></property><property>
>>>>>>>>>>>>>> <name>yarn.scheduler.capacity.root.capacity</name>
>>>>>>>>>>>>>> <value>100</value></property><property>
>>>>>>>>>>>>>> <name>yarn.scheduler.capacity.root.production.capacity</name>
>>>>>>>>>>>>>> <value>70</value></property>
<property>
>>>>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.capacity</name>
>>>>>>>>>>>>>> <value>30</value></property><property>
>>>>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.a.capacity</name>
>>>>>>>>>>>>>> <value>30</value></property><property>
>>>>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.b.capacity</name>
>>>>>>>>>>>>>> <value>30</value></property>
<property>
>>>>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.c.capacity</name>
>>>>>>>>>>>>>> <value>40</value></property><property>
 <name>yarn.scheduler.capacity.root.
>>>>>>>>>>>>>> production.acl_submit_applications</name>
>>>>>>>>>>>>>> <value>yarn,mapred</value></property><property>
   <value>userb</value>
>>>>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.a.acl_submit_applications</name>
>>>>>>>>>>>>>> </property><property>
>>>>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.b.acl_submit_applications</name>
>>>>>>>>>>>>>> <value>userb</value></property><property>
>>>>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.c.acl_submit_applications</name>
>>>>>>>>>>>>>> <value>userc</value></property></configuration>*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> *Regards*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  *Munna*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual
or
>>>>>>> entity to which it is addressed and may contain information that
is
>>>>>>> confidential, privileged and exempt from disclosure under applicable
law.
>>>>>>> If the reader of this message is not the intended recipient,
you are hereby
>>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>>> disclosure or forwarding of this communication is strictly prohibited.
If
>>>>>>> you have received this communication in error, please contact
the sender
>>>>>>> immediately and delete it from your system. Thank You.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Regards*
>>>>>>
>>>>>> *Munna*
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Regards*
>>>>>
>>>>> *Munna*
>>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>>
>>>
>>>
>>>
>>> --
>>> *Regards*
>>>
>>> *Munna*
>>>
>>
>>
>>
>> --
>> *Regards*
>>
>> *Munna*
>>
>
>
>
> --
>    * Olivier Renault *       Solution Engineer
> ------------------------------
>
>     Phone:        +44 7500 933 036
>   Email:      orenault@hortonworks.com
>   Website:   http://www.hortonworks.com/
>
>       * Follow Us: *
> <http://facebook.com/hortonworks/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <http://twitter.com/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <http://www.linkedin.com/company/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
>  [image: photo]
>
>   Latest From Our Blog:  SAP HANA + Hadoop: A Perfect Match <http://hortonworks.com/blog/sap-hana-hadoop-a-perfect-match/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 
*Regards*

*Munna*

Mime
View raw message