hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3341) Enhance logging of initalized queue limit values
Date Wed, 09 Nov 2011 20:37:53 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147266#comment-13147266
] 

jiraposter@reviews.apache.org commented on MAPREDUCE-3341:
----------------------------------------------------------



bq.  On 2011-11-08 17:27:47, Robert Evans wrote:
bq.  > Mostly looks good, just one minor nit.

Thanks Bobby for noticing that! Below is an example log output. I will upload a new patch
removing the unnecssary whitespace.

2011-11-09 20:20:19,309 INFO  capacity.LeafQueue (LeafQueue.java:setupQueueConfigs(250)) -
Initializing default
capacity = 0.4[ = (float) configuredCapacity / 100 ]
asboluteCapacity = 0.4[ = parentAbsoluteCapacity * capacity ]
maxCapacity = -0.01[ = configuredMaxCapacity ]
absoluteMaxCapacity = 3.4028235E38[ = Float.MAX_VALUE if maximumCapacity undefined,   = (parentAbsoluteCapacity
* maximumCapacity) / 100 otherwise ]
userLimit = 100[ = configuredUserLimit ]
userLimitFactor = 1.0[ = configuredUserLimitFactor ]
maxApplications = 4000[ = (int)(configuredMaximumSystemApplications * absoluteCapacity) ]
maxApplicationsPerUser = 4000[ = (int)(maxApplications * (userLimit / 100.0f) * userLimitFactor)
]
maxActiveApplications = 1[ = max(          (int)((clusterResourceMemory / (float)DEFAULT_AM_RESOURCE)
*           maxAMResourcePercent * absoluteCapacity),    1) ]
maxActiveApplicationsPerUser = 1[ = (int)(maxActiveApplications * (userLimit / 100.0f) * userLimitFactor)
]
utilization = 0.0[ = usedResourcesMemory / queueLimit ]
usedCapacity = 0.0[ = usedResourcesMemory / (clusterResourceMemory * capacity) ]
maxAMResourcePercent = 0.1[ = configuredMaximumAMResourcePercent ]
minimumAllocationFactor = 0.9[ = (float)(maximumAllocationMemory - minimumAllocationMemory)
/ maximumAllocationMemory ]
numContainers = 0[ = currentNumContainers ]
state = RUNNING[ = configuredState ]
acls = ADMINISTER_QUEUE:*SUBMIT_APPLICATIONS:*[ = configuredAcls ]


- Anupam


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2775/#review3112
-----------------------------------------------------------


On 2011-11-09 20:34:10, Anupam Seth wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2775/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-11-09 20:34:10)
bq.  
bq.  
bq.  Review request for Robert Evans and Jonathan Eagles.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  Enhance logging for queues in capacity scheduler
bq.  
bq.  
bq.  This addresses bug MAPREDUCE-3341.
bq.      https://issues.apache.org/jira/browse/MAPREDUCE-3341
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
1198816 
bq.  
bq.  Diff: https://reviews.apache.org/r/2775/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Anupam
bq.  
bq.


                
> Enhance logging of initalized queue limit values
> ------------------------------------------------
>
>                 Key: MAPREDUCE-3341
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3341
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Anupam Seth
>            Assignee: Anupam Seth
>         Attachments: MAPREDUCE-3341-branch_0_23.patch
>
>
> Currently the RM log shows only a partial set of the limits that are configured when
a queue is initialized / reinitialized.
> For example, this is what is currently shown in the RM log for an initialized queue:
> # <datestamp> INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
Initializing
> default, capacity=0.25, asboluteCapacity=0.25, maxCapacity=25.0, asboluteMaxCapacity=0.25,
userLimit=100,
> userLimitFactor=20.0, maxApplications=2500, maxApplicationsPerUser=50000, state=RUNNING,
> acls=ADMINISTER_QUEUE:*SUBMIT_JOB:*ADMINISTER_JOBS:*
> Breaking down the line above, shows: 
> capacity=0.25
> asboluteCapacity=0.25
> maxCapacity=25.0
> asboluteMaxCapacity=0.25
> userLimit=100
> userLimitFactor=20.0
> maxApplications=2500
> maxApplicationsPerUser=50000
> It might be nice if we could include more information such as maxActiveApplications,
maxActiveApplicationsPerUser, utilization, and usedCapacity along with information on how
each of these is computed (i.e. formulae used) (Thanks to Phil Su for requesting this).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message