hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Payne (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-3897) capacity scheduler - maxActiveApplicationsPerUser calculation can be wrong
Date Wed, 29 Feb 2012 22:47:57 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-3897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Eric Payne updated MAPREDUCE-3897:

    Attachment: MAPREDUCE-3897-1.txt

Changed calculation for maxActiveApplications.

Old: maxActiveApplications = Max((((total cluster mem) / minimumAllocation) * maxAMResourcePercent
* absoluteMaxCapacity),1)

New: maxActiveApplications = Max((((total cluster mem) / minimumAllocation) * maxAMResourcePercent
* absoluteCapacity),1)

By using absoluteCapacity rather than absoluteMaxCapacity, the result is based on a per-queue
capacity and prevents a small queue from hitting the deadlock condition of being filled with
all AMs.

This fix will be further enhanced when it is possible to specify maxAMResourcePercent on a
per-queue basis (see MAPREDUCE:3893)
> capacity scheduler - maxActiveApplicationsPerUser calculation can be wrong
> --------------------------------------------------------------------------
>                 Key: MAPREDUCE-3897
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3897
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Thomas Graves
>            Assignee: Eric Payne
>            Priority: Critical
>         Attachments: MAPREDUCE-3897-1.txt
> The capacity scheduler calculates the maxActiveApplications and the maxActiveApplicationsPerUser
based on the config yarn.scheduler.capacity.maximum-applications or default 10000.  
> MaxActiveApplications = max ( ceil ( clusterMemory/minAllocation * maxAMResource% * absoluteMaxCapacity),
> MaxActiveAppsPerUser = max( ceil (maxActiveApplicationsComputedAbove * (userLimit%/100)
* userLimitFactor), 1) 
> maxActiveApplications is already multiplied by the queue absolute MAXIMUM capacity, so
if max capacity > capacity and if you have user limit factor 1 (which is the default) and
only 1 user is running, that user will not be allowed to use over the queue capacity, so having
it relative to MAX capacity doesn't make sense.  That user could easily end up in a deadlock
and all its space used by application masters.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message