hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-7739) Revisit scheduler resource normalization behavior for max allocation
Date Fri, 12 Jan 2018 00:39:00 GMT
Wangda Tan created YARN-7739:

             Summary: Revisit scheduler resource normalization behavior for max allocation
                 Key: YARN-7739
                 URL: https://issues.apache.org/jira/browse/YARN-7739
             Project: Hadoop YARN
          Issue Type: Bug
            Reporter: Wangda Tan
            Priority: Critical

Currently, YARN Scheduler normalizes requested resource based on the maximum allocation derived
from configured maximum allocation and maximum registered node resources. Basically, the scheduler
will silently cap asked resource by maximum allocation.

This could cause issues for applications, for example, a Spark job which needs 12 GB memory
to run, however in the cluster, registered NMs have at most 8 GB mem on each node. So scheduler
allocates 8GB memory container to the requested application.

Once app receives containers from RM, if it doesn't double check allocated resources, it will
lead to OOM and hard to debug because scheduler silently caps maximum allocation.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message