hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-521) Augment AM - RM client module to be able to request containers only at specific locations
Date Thu, 11 Jul 2013 00:33:48 GMT

    [ https://issues.apache.org/jira/browse/YARN-521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705283#comment-13705283

Bikas Saha commented on YARN-521:

Shouldnt this also check that the inferred rack1 has relaxLocality set to false?
+    ContainerRequest bothLevelRequest =
+        new ContainerRequest(capability, new String[] {"host3", "host4"},
+            new String[] {NetworkTopology.DEFAULT_RACK, "/otherrack"},
+            Priority.newInstance(3), 4, false);
+    client.addContainerRequest(bothLevelRequest);
+    verifyResourceRequest(client, bothLevelRequest, ResourceRequest.ANY, false);
+    verifyResourceRequest(client, bothLevelRequest, NetworkTopology.DEFAULT_RACK,
+        true);
+    verifyResourceRequest(client, bothLevelRequest, "/otherrack",
+        true);
+    verifyResourceRequest(client, bothLevelRequest, "host3", true);
+    verifyResourceRequest(client, bothLevelRequest, "host4", true);

The RM allows relaxLocality to be re-enabled on a location and priority. This allows users
to change their minds about strict locality if they are unable to get those containers for
a long time. The logic in the patch does not allow this to happen. In fact testDifferentLocalityRelaxationSamePriority()
shows that it is not possible to do this with the current patch.

testDifferentLocalityRelaxationSamePriority() looks to have been written to test for the case
where host1 is specific resulting in rack1 to be false. Then host3 in rack1 cannot be made
non-specific since the flag on rack1 will be inconsistent. It would be more clear if the second
request had a different host.

testLocalityRelaxationDifferentLevels() tests that specific host cannot be mixed with non-specific
rack for the rack of that host. Can specific host be mixed with specific rack for the same
rack as the host? Probably not.

Can specific node host1 be mixed with specific rack on a different rack (rack2)? If yes, if
a container is allocated on host2 in rack2, then what prevents getMachingRequests(*) from
returning the requests for host1 and rack2 instead of only the request for rack2?

Now that we have a client and server pieces done, has this been tested with a real cluster
to see that it works for in practice?

It would really help if we had a clear document (even javadoc) that clarifies what is legal
and what is not. This will really help in code review, it will make sure we have better test
coverage and in general will help users understand how to use the API. I am concerned that
we are implementing something pretty subtle and we need to get it right logically before implementing
the code. I havent even started thinking about how this interacts with the blacklisting logic
that we have added to the RM. How specific requests interact with blacklist on the RM. And
how both interact on the AMRMClient. And how both interact across RM and AMRMClient.

Minor nits.
How about "inferredRacks" instead of "filledInRacks"?
How about JAVA standard library Collections.singletonList() instead of invoking google common
> Augment AM - RM client module to be able to request containers only at specific locations
> -----------------------------------------------------------------------------------------
>                 Key: YARN-521
>                 URL: https://issues.apache.org/jira/browse/YARN-521
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: api
>    Affects Versions: 2.0.3-alpha
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
>         Attachments: YARN-521-1.patch, YARN-521-2.patch, YARN-521-2.patch, YARN-521.patch
> When YARN-392 and YARN-398 are completed, it would be good for AMRMClient to offer an
easy way to access their functionality

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message