hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-193) Scheduler.normalizeRequest does not account for allocation requests that exceed maximumAllocation limits
Date Tue, 02 Apr 2013 17:57:16 GMT

    [ https://issues.apache.org/jira/browse/YARN-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620066#comment-13620066
] 

Bikas Saha commented on YARN-193:
---------------------------------

Can we check that we are getting the expected exception and not some other one?
{code}
+    try {
+      rmService.submitApplication(submitRequest);
+      Assert.fail("Application submission should fail because");
+    } catch (YarnRemoteException e) {
+      // Exception is expected
+    }
+  }
{code}

Setting the same config twice? In second set, why not use a -ve value instead of the DISABLE
value? Its not clear whether we want to disable check or set a -ve value. same for others.
{code}
+    conf.setInt(YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, 0);
+    conf.setInt(YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
+        ResourceCalculator.DISABLE_RESOURCELIMIT_CHECK);
+    try {
+      resourceManager.init(conf);
+      fail("Exception is expected because the min memory allocation is" +
+          " non-positive.");
+    } catch (YarnException e) {
+      // Exception is expected.
{code}

Lets also add a test for case when memory is more than max. Normalize should always reduce
that to max. Same for DRF
{code}
+    // max is not a multiple of min
+    maxResource = Resources.createResource(maxMemory - 10, 0);
+    ask.setCapability(Resources.createResource(maxMemory - 100));
+    // multiple of minMemory > maxMemory, then reduce to maxMemory
+    SchedulerUtils.normalizeRequest(ask, resourceCalculator, null,
+        minResource, maxResource);
+    assertEquals(maxResource.getMemory(), ask.getCapability().getMemory());
   }
{code}

Rename testAppSubmitError() to show that its testing invalid resource request?

TestAMRMClient. Why is this change needed?
{code}
+    amResource.setMemory(
+        YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
+    amContainer.setResource(amResource);
{code}

Dont we need to throw?
{code}
+      } catch (InvalidResourceRequestException e) {
+        LOG.info("Resource request was not able to be alloacated for" +
+            " application attempt " + appAttemptId + " because it" +
+            " failed to pass the validation. " + e.getMessage());
+        RPCUtil.getRemoteException(e);
+      }
{code}

typo
{code}
+    // validate scheduler vcors allocation setting
{code}

This will need to be rebased after YARN-382 which I am going to commit shortly.

I am fine with requiring that a max allocation limit be set. We should also make sure that
max allocation from config can be matched by at least 1 machine in the cluster. That should
be a different jira.

IMO, Normalization should be called only inside the scheduler. It is an artifact of the scheduler
logic. Nothing in the RM requires resources to be normalized to a multiple of min. Only the
scheduler needs it to makes its life easier and it could choose to not do so.


                
> Scheduler.normalizeRequest does not account for allocation requests that exceed maximumAllocation
limits 
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-193
>                 URL: https://issues.apache.org/jira/browse/YARN-193
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.0.2-alpha, 3.0.0
>            Reporter: Hitesh Shah
>            Assignee: Zhijie Shen
>         Attachments: MR-3796.1.patch, MR-3796.2.patch, MR-3796.3.patch, MR-3796.wip.patch,
YARN-193.10.patch, YARN-193.11.patch, YARN-193.4.patch, YARN-193.5.patch, YARN-193.6.patch,
YARN-193.7.patch, YARN-193.8.patch, YARN-193.9.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message