hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Munna <munnava...@gmail.com>
Subject Re: Capacity Scheduler Issue
Date Thu, 28 Nov 2013 12:40:23 GMT
Hi,


I think there is no solution on above issue, so i'll move to fair scheduler.

Thanks to all...

Munna


On Thu, Nov 28, 2013 at 9:11 AM, Munna <munnavarsk@gmail.com> wrote:

> I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
>
>
> what is the best value?
>
> Tx,
> Munna
>
>
> On Thu, Nov 28, 2013 at 12:34 AM, Jian He <jhe@hortonworks.com> wrote:
>
>> The log shows the both queues are properly picked up by the RM.
>> If the problem is that your submitted application is not able to run, you
>> may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
>> this controls the max number of concurrently running AMs in the cluster.
>>
>> Jian
>>
>>
>> On Wed, Nov 27, 2013 at 9:42 AM, Munna <munnavarsk@gmail.com> wrote:
>>
>>> Hi Flocks,
>>>
>>>
>>>
>>> Since, last two days I am about to configure Capacity Scheduler. Here,
>>> how I have struggling L….
>>>
>>>
>>>
>>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4
>>> Queue’s as per the Capacity Scheduler Documentation and those queues shown
>>> in RM UI.
>>>
>>>
>>>
>>> After configuration I tried to run Jobs, Cap Scheduler not identified
>>> that queue’s. where I have check queues list with “mapred queue –list”,
>>> which showing all configured Q’s.
>>>
>>>
>>>
>>> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
>>> idea about that, based on his views I dig more.
>>>
>>>
>>>
>>> After I went to all the RM log, Cap Scheduler initiating only default
>>> “default”, I have tested with *default queue* it works for me. And I
>>> have created one more queue called “dev”, in this Queue User unable to run
>>> the jobs and its unable to identifying users Queue.
>>>
>>>
>>>
>>> I have attached Cap Scheduler configuration file for your information.
>>> Some O/P for ur information.
>>>
>>>
>>>
>>> *[user@host ~]$ mapred queue -list*
>>>
>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>>
>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>>
>>> *======================*
>>>
>>> *Queue Name : dev*
>>>
>>> *Queue State : running*
>>>
>>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>>> CurrentCapacity: 0.0*
>>>
>>> *======================*
>>>
>>> *Queue Name : default*
>>>
>>> *Queue State : running*
>>>
>>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0,
>>> CurrentCapacity: 0.0*
>>>
>>>
>>>
>>> *RM log Scheduler loading info:*
>>>
>>> 2013-11-27 08:54:58,521 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>>> ADMINISTER_QUEUE:
>>>
>>> 2013-11-27 08:54:58,521 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>> Initialized parent-queue root name=root, fullname=root
>>>
>>> 2013-11-27 08:54:58,534 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>> default*
>>>
>>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>>
>>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>>
>>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>>
>>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>
>>> userLimit = 100 [= configuredUserLimit ]
>>>
>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>
>>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or
>>> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>
>>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>>> 100.0f) * userLimitFactor) ]
>>>
>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>> ]
>>>
>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>
>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>
>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>> absoluteCapacity)]
>>>
>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>> clusterResourceMemory]
>>>
>>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent
>>> ]
>>>
>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>
>>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent
>>> ]
>>>
>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>
>>> numContainers = 0 [= currentNumContainers ]
>>>
>>> state = RUNNING [= configuredState ]
>>>
>>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE:  [=
>>> configuredAcls ]
>>>
>>>
>>>
>>> 2013-11-27 08:54:58,534 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>>
>>> 2013-11-27 08:54:58,543 INFO
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing
>>> dev*
>>>
>>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>>
>>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>>
>>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>>
>>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>
>>> userLimit = 100 [= configuredUserLimit ]
>>>
>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>
>>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or
>>> (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>
>>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>>> 100.0f) * userLimitFactor) ]
>>>
>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>> ]
>>>
>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>
>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>
>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>> absoluteCapacity)]
>>>
>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>> clusterResourceMemory]
>>>
>>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent
>>> ]
>>>
>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>
>>> numContainers = 0 [= currentNumContainers ]
>>>
>>> state = RUNNING [= configuredState ]
>>>
>>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE:  [=
>>> configuredAcls ]
>>>
>>>
>>>
>>> Can you guys please confirm, did I miss anything on configurations part
>>> or is there any bug persist on 2.0.0?
>>>
>>>
>>>
>>> Thanks
>>>
>>> Munna
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
>
> --
> *Regards*
>
> *Munna*
>



-- 
*Regards*

*Munna*

Mime
View raw message