hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Brokaw (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-7273) Ability for YARN applications to request lower container size below yarn.scheduler.minimum-allocation-mb
Date Fri, 29 Sep 2017 14:27:00 GMT
Scott Brokaw created YARN-7273:

             Summary: Ability for YARN applications to request lower container size below
                 Key: YARN-7273
                 URL: https://issues.apache.org/jira/browse/YARN-7273
             Project: Hadoop YARN
          Issue Type: Improvement
            Reporter: Scott Brokaw
            Priority: Minor

Currently, a YARN application must request a container size that is between the yarn.scheduler.minimum-allocation-mb
and yarn.scheduler.maximum-allocation-mb else the request has to be raised by the Application
to the yarn.scheduler.minimum-allocation-mb

yarn.scheduler.minimum-allocation-mb < Requested Container Size > yarn.scheduler.maximum-allocation-mb

This improvement would either add a new parameter such as yarn.scheduler.default-allocation-mb
or treat yarn.scheduler.minimum-allocation-mb instead as a global default size if no container
size is requested.  This would allow a particular YARN application to request a smaller amount
of resources if it deems it can run with resources below the minimum allocation dictated by
YARN.  This allows a significant amount of resources to be saved as otherwise a YARN application
that spawns many containers can end up consuming many more resources then it needs.

The same idea can be applied to yarn.scheduler.minimum-allocation-vcores.

The argument can be made that you should just set the minimum values lower to start with,
but the idea is that YARN should support a global default value for applications that don't
set it, but allow other applications (if they so request) to be granted smaller containers.
 The "problem" quickly compounds the more containers an application spins up and with memory
being a valuable resource in the cluster, this improvement is looking to maximize the available
resources given an application is "willing and able" to run with a smaller size.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

View raw message