hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jian He <...@hortonworks.com>
Subject Re: Capacity Scheduler Issue
Date Wed, 27 Nov 2013 19:04:54 GMT
The log shows the both queues are properly picked up by the RM.
If the problem is that your submitted application is not able to run, you
may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
this controls the max number of concurrently running AMs in the cluster.

Jian


On Wed, Nov 27, 2013 at 9:42 AM, Munna <munnavarsk@gmail.com> wrote:

> Hi Flocks,
>
>
>
> Since, last two days I am about to configure Capacity Scheduler. Here, how
> I have struggling L….
>
>
>
> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4
> Queue’s as per the Capacity Scheduler Documentation and those queues shown
> in RM UI.
>
>
>
> After configuration I tried to run Jobs, Cap Scheduler not identified that
> queue’s. where I have check queues list with “mapred queue –list”, which
> showing all configured Q’s.
>
>
>
> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
> idea about that, based on his views I dig more.
>
>
>
> After I went to all the RM log, Cap Scheduler initiating only default
> “default”, I have tested with *default queue* it works for me. And I have
> created one more queue called “dev”, in this Queue User unable to run the
> jobs and its unable to identifying users Queue.
>
>
>
> I have attached Cap Scheduler configuration file for your information.
> Some O/P for ur information.
>
>
>
> *[user@host ~]$ mapred queue -list*
>
> *13/11/27 09:26:38 INFO service.AbstractService:
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>
> *13/11/27 09:26:38 INFO service.AbstractService:
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>
> *======================*
>
> *Queue Name : dev*
>
> *Queue State : running*
>
> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
> CurrentCapacity: 0.0*
>
> *======================*
>
> *Queue Name : default*
>
> *Queue State : running*
>
> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, CurrentCapacity:
> 0.0*
>
>
>
> *RM log Scheduler loading info:*
>
> 2013-11-27 08:54:58,521 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
> ADMINISTER_QUEUE:
>
> 2013-11-27 08:54:58,521 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Initialized parent-queue root name=root, fullname=root
>
> 2013-11-27 08:54:58,534 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
> default*
>
> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>
> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>
> maxCapacity = 1.0 [= configuredMaxCapacity ]
>
> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>
> userLimit = 100 [= configuredUserLimit ]
>
> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>
> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or
> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>
> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
> 100.0f) * userLimitFactor) ]
>
> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
> ]
>
> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>
> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
> (userLimit / 100.0f) * userLimitFactor),1) ]
>
> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
> absoluteCapacity)]
>
> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
>
> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>
> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / maximumAllocationMemory ]
>
> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>
> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / maximumAllocationMemory ]
>
> numContainers = 0 [= currentNumContainers ]
>
> state = RUNNING [= configuredState ]
>
> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE:  [=
> configuredAcls ]
>
>
>
> 2013-11-27 08:54:58,534 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>
> 2013-11-27 08:54:58,543 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
> dev*
>
> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>
> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>
> maxCapacity = 0.5 [= configuredMaxCapacity ]
>
> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>
> userLimit = 100 [= configuredUserLimit ]
>
> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>
> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or
> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>
> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
> 100.0f) * userLimitFactor) ]
>
> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
> ]
>
> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>
> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
> (userLimit / 100.0f) * userLimitFactor),1) ]
>
> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
> absoluteCapacity)]
>
> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
>
> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
>
> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / maximumAllocationMemory ]
>
> numContainers = 0 [= currentNumContainers ]
>
> state = RUNNING [= configuredState ]
>
> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE:  [= configuredAcls
> ]
>
>
>
> Can you guys please confirm, did I miss anything on configurations part or
> is there any bug persist on 2.0.0?
>
>
>
> Thanks
>
> Munna
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message