hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Munna <munnava...@gmail.com>
Subject Re: Capacity Scheduler Issue
Date Thu, 28 Nov 2013 03:41:11 GMT
I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*


what is the best value?

Tx,
Munna


On Thu, Nov 28, 2013 at 12:34 AM, Jian He <jhe@hortonworks.com> wrote:

> The log shows the both queues are properly picked up by the RM.
> If the problem is that your submitted application is not able to run, you
> may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
> this controls the max number of concurrently running AMs in the cluster.
>
> Jian
>
>
> On Wed, Nov 27, 2013 at 9:42 AM, Munna <munnavarsk@gmail.com> wrote:
>
>> Hi Flocks,
>>
>>
>>
>> Since, last two days I am about to configure Capacity Scheduler. Here,
>> how I have struggling L….
>>
>>
>>
>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4
>> Queue’s as per the Capacity Scheduler Documentation and those queues shown
>> in RM UI.
>>
>>
>>
>> After configuration I tried to run Jobs, Cap Scheduler not identified
>> that queue’s. where I have check queues list with “mapred queue –list”,
>> which showing all configured Q’s.
>>
>>
>>
>> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
>> idea about that, based on his views I dig more.
>>
>>
>>
>> After I went to all the RM log, Cap Scheduler initiating only default
>> “default”, I have tested with *default queue* it works for me. And I
>> have created one more queue called “dev”, in this Queue User unable to run
>> the jobs and its unable to identifying users Queue.
>>
>>
>>
>> I have attached Cap Scheduler configuration file for your information.
>> Some O/P for ur information.
>>
>>
>>
>> *[user@host ~]$ mapred queue -list*
>>
>> *13/11/27 09:26:38 INFO service.AbstractService:
>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>
>> *13/11/27 09:26:38 INFO service.AbstractService:
>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>
>> *======================*
>>
>> *Queue Name : dev*
>>
>> *Queue State : running*
>>
>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>> CurrentCapacity: 0.0*
>>
>> *======================*
>>
>> *Queue Name : default*
>>
>> *Queue State : running*
>>
>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, CurrentCapacity:
>> 0.0*
>>
>>
>>
>> *RM log Scheduler loading info:*
>>
>> 2013-11-27 08:54:58,521 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>> ADMINISTER_QUEUE:
>>
>> 2013-11-27 08:54:58,521 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>> Initialized parent-queue root name=root, fullname=root
>>
>> 2013-11-27 08:54:58,534 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>> default*
>>
>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>
>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>
>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>
>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>
>> userLimit = 100 [= configuredUserLimit ]
>>
>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>
>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or
>> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>
>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>> 100.0f) * userLimitFactor) ]
>>
>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>> ]
>>
>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>
>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>
>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>> absoluteCapacity)]
>>
>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
>>
>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>>
>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>> minimumAllocationMemory) / maximumAllocationMemory ]
>>
>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>>
>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>> minimumAllocationMemory) / maximumAllocationMemory ]
>>
>> numContainers = 0 [= currentNumContainers ]
>>
>> state = RUNNING [= configuredState ]
>>
>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE:  [=
>> configuredAcls ]
>>
>>
>>
>> 2013-11-27 08:54:58,534 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>
>> 2013-11-27 08:54:58,543 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>> dev*
>>
>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>
>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>
>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>
>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>
>> userLimit = 100 [= configuredUserLimit ]
>>
>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>
>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or
>> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>
>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>> 100.0f) * userLimitFactor) ]
>>
>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>> ]
>>
>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>
>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>
>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>> absoluteCapacity)]
>>
>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
>>
>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>>
>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>> minimumAllocationMemory) / maximumAllocationMemory ]
>>
>> numContainers = 0 [= currentNumContainers ]
>>
>> state = RUNNING [= configuredState ]
>>
>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE:  [=
>> configuredAcls ]
>>
>>
>>
>> Can you guys please confirm, did I miss anything on configurations part
>> or is there any bug persist on 2.0.0?
>>
>>
>>
>> Thanks
>>
>> Munna
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.




-- 
*Regards*

*Munna*

Mime
View raw message