hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brahma Reddy Battula (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4678) Cluster used capacity is > 100 when container reserved
Date Mon, 08 Feb 2016 05:18:39 GMT

    [ https://issues.apache.org/jira/browse/YARN-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136594#comment-15136594
] 

Brahma Reddy Battula commented on YARN-4678:
--------------------------------------------

 *As reserved memory is include in the used resources. queue capacity > 100* ..Please let
me know any thoughts on this..

2016-02-22 01:15:09,359 | INFO  | ResourceManager Event Processor | completedContainer queue=root
usedCapacity=0.9583333 absoluteUsedCapacity=0.9583333  *{color:red}used=<memory:23552,
vCores:7> cluster=<memory:24576, vCores:24>{color}*  | ParentQueue.java:631
2016-02-22 01:15:09,359 | INFO  | ResourceManager Event Processor | Re-sorting completed queue:
root.QueueA stats: QueueA: capacity=0.4, absoluteCapacity=0.4, usedResources=<memory:17920,
vCores:5>, usedCapacity=1.8229909, absoluteUsedCapacity=0.7291667, numApps=1, numContainers=5
| ParentQueue.java:648
2016-02-22 01:15:09,359 | INFO  | ResourceManager Event Processor | Application attempt appattempt_1456061422894_0003_000001
released container container_e05_1456061422894_0003_01_004504 on  *{color:blue}node: host:
vm-3:26009{color}*  #containers=1 available=<memory:6656, vCores:7> used=<memory:1536,
vCores:1> with event: FINISHED | CapacityScheduler.java:1452
2016-02-22 01:15:09,359 | INFO  | ResourceManager Event Processor | Trying to fulfill reservation
for application application_1456061422894_0003 on node: vm-3:26009 | CapacityScheduler.java:1110
2016-02-22 01:15:09,359 | INFO  | ResourceManager Event Processor | Application application_1456061422894_0003
*{color:blue}unreserved  on node host: vm-3:26009{color}*  #containers=1 available=<memory:6656,
vCores:7> used=<memory:1536, vCores:1>, currently has 0 at priority 20; currentReservation
<memory:0, vCores:0> on node-label= | FiCaSchedulerApp.java:229
2016-02-22 01:15:09,359 | INFO  | ResourceManager Event Processor | container_e05_1456061422894_0003_01_004505
Container Transitioned from NEW to ALLOCATED | RMContainerImpl.java:419
2016-02-22 01:15:09,359 | INFO  | ResourceManager Event Processor | Assigned container container_e05_1456061422894_0003_01_004505
of capacity <memory:4096, vCores:1> on  *{color:blue}host vm-3:26009{color}* , which
has 2 containers, <memory:5632, vCores:2> used and <memory:2560, vCores:6> available
after allocation | SchedulerNode.java:154
2016-02-22 01:15:09,359 | INFO  | ResourceManager Event Processor | assignedContainer application
attempt=appattempt_1456061422894_0003_000001 container=Container: [ContainerId: container_e05_1456061422894_0003_01_004505,
NodeId: vm-3:26009, NodeHttpAddress: vm-3:26010, Resource: <memory:4096, vCores:1>,
Priority: 20, Token: null, ] queue=QueueA: capacity=0.4, absoluteCapacity=0.4, usedResources=<memory:17920,
vCores:5>, usedCapacity=1.8229909, absoluteUsedCapacity=0.7291667, numApps=1, numContainers=5
clusterResource=<memory:24576, vCores:24> | LeafQueue.java:1731
2016-02-22 01:15:09,360 | INFO  | ResourceManager Event Processor | container_e05_1456061422894_0003_01_004508
Container Transitioned from *{color:blue}NEW to RESERVED{color}*  | RMContainerImpl.java:419
2016-02-22 01:15:09,360 | INFO  | ResourceManager Event Processor |  *Reserved container 
application=application_1456061422894_0003 resource=<memory:4096, vCores:1>*  queue=QueueA:
capacity=0.4, absoluteCapacity=0.4, usedResources=<memory:17920, vCores:5>, usedCapacity=1.8229909,
absoluteUsedCapacity=0.7291667, numApps=1, numContainers=5 usedCapacity=1.8229909 absoluteUsedCapacity=0.7291667
used=<memory:17920, vCores:5> cluster=<memory:24576, vCores:24> | LeafQueue.java:1764
2016-02-22 01:15:09,360 | INFO  | ResourceManager Event Processor | Re-sorting assigned queue:
root.QueueA stats: QueueA: capacity=0.4, absoluteCapacity=0.4, usedResources=<memory:22016,
vCores:6>, usedCapacity=2.2396743, absoluteUsedCapacity=0.8958333, numApps=1, numContainers=6
| ParentQueue.java:585
2016-02-22 01:15:09,360 | INFO  | ResourceManager Event Processor |  *assignedContainer queue=root
{color:red}usedCapacity=1.125 absoluteUsedCapacity=1.125 
used=<memory:27648, vCores:8> 
cluster=<memory:24576, vCores:24>{color}*  | ParentQueue.java:465

> Cluster used capacity is > 100 when container reserved 
> -------------------------------------------------------
>
>                 Key: YARN-4678
>                 URL: https://issues.apache.org/jira/browse/YARN-4678
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Brahma Reddy Battula
>            Assignee: Brahma Reddy Battula
>
>  *Scenario:* 
> * Start cluster with Three NM's each having 8GB (cluster memory:24GB).
> * Configure queues with elasticity and userlimitfactor=10.
> * disable pre-emption.
> * run two job with different priority in different queue at the same time
> ** yarn jar hadoop-mapreduce-examples-2.7.2.jar pi -Dyarn.app.priority=LOW -Dmapreduce.job.queuename=QueueA
-Dmapreduce.map.memory.mb=4096 -Dyarn.app.mapreduce.am.resource.mb=1536 -Dmapreduce.job.reduce.slowstart.completedmaps=1.0
10 1000000000000
> ** yarn jar hadoop-mapreduce-examples-2.7.2.jar pi -Dyarn.app.priority=HIGH -Dmapreduce.job.queuename=QueueB
-Dmapreduce.map.memory.mb=4096 -Dyarn.app.mapreduce.am.resource.mb=1536 3 1000000000000
> * observe the cluster capacity which was used in RM web UI



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message