hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-193) Scheduler.normalizeRequest does not account for allocation requests that exceed maximumAllocation limits
Date Tue, 02 Apr 2013 16:21:16 GMT

    [ https://issues.apache.org/jira/browse/YARN-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13619957#comment-13619957

Zhijie Shen commented on YARN-193:

I am not sure if we should allow disabling of the max memory and max vcores setting. Was it
supported earlier or does the patch introduce this support?

Yes, the patch introduces the support. It is already there in your previous patch. I inherit
it and and some description in yarn-default.xml. I'm fine with whether the function need to
be supported or not. One risk I can image if the function is supported is that AM memory can
exceeds "yarn.nodemanager.resource.memory-mb" when DISABLE_RESOURCELIMIT_CHECK is set. Then,
the problem described in YARN-389 will occur.

Question - should normalization of resource requests be done inside the scheduler or in the
ApplicationMasterService itself which handles the allocate call?
I think it should be better to do normalization outside allocate, because allocate is not
only called in ApplicationMasterService and it is not necessary that normalize is called every
time when allocate is called. For example, RMAppAttemptImpl#ScheduleTransition#transition
doesn't require to do normalization because the resource has been validated during the submission
stage. For another example, RMAppAttemptImpl#AMContainerAllocatedTransition#transition supplies
an empty ask. 

Unrelated to this patch but when throwing/logging errors related to configs, we should always
point to the configuration property to let the user know which property needs to be changed.
Please file a separate jira for the above.
I'll do that, and modify the log information when exception is thrown in this patch.

For InvalidResourceRequestException, missing javadocs for class description.
I'll add the description.

If maxMemory or maxVcores is set to -1, what will happen when normalize() is called?
The normalized value has not upper bound.

> Scheduler.normalizeRequest does not account for allocation requests that exceed maximumAllocation
> ---------------------------------------------------------------------------------------------------------
>                 Key: YARN-193
>                 URL: https://issues.apache.org/jira/browse/YARN-193
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.0.2-alpha, 3.0.0
>            Reporter: Hitesh Shah
>            Assignee: Zhijie Shen
>         Attachments: MR-3796.1.patch, MR-3796.2.patch, MR-3796.3.patch, MR-3796.wip.patch,
YARN-193.10.patch, YARN-193.11.patch, YARN-193.4.patch, YARN-193.5.patch, YARN-193.6.patch,
YARN-193.7.patch, YARN-193.8.patch, YARN-193.9.patch

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message