hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-5124) Modify AMRMClient to set the ExecutionType in the ResourceRequest
Date Tue, 07 Jun 2016 00:47:21 GMT

     [ https://issues.apache.org/jira/browse/YARN-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Arun Suresh updated YARN-5124:
    Attachment: YARN-5124.010.patch

Uploading patch addressing most of [~curino]'s feedback :

bq. Can we wrap this datastructure in some object? Map<Priority, Map<String, Map<ExecutionType,
TreeMap<Resource, ResourceRequestInfo>>>> it is starting to get a bit too complicated
to follow, and we might eventually want to change representations.
I agree, it looks ugly, but even if we pull it up in a new class, we would still see it. In
the latest patch I alleviated the issue by somewhat emulating C++ typedefs as follows:
# I create *typed* subclasses of HashMap classes like so:
  class CapabilityMap extends TreeMap<Resource, ResourceRequestInfo> {}
  class ExecutionTypeMap extends HashMap<ExecutionType, CapabilityMap> {}
  class LocationMap extends HashMap<String, ExecutionTypeMap> {}
  class RemoteRequestsTable extends HashMap<Priority, LocationMap> {}
# The remoteRequestsTable then looks like this:
final RemoteRequestsTable remoteRequestsTable = new RemoteRequestsTable();

bq. Do you need the new constructor AMRMClientImpl(ApplicationMasterProtocol protocol)? Can't
you use mockito for the conf in tests and pass the ApplicationMasterProtocol that way?
I can't use mockito, since I actually have a real AMRMClientImpl object. This is a functional
test that uses an actual MiniYARNCluster. So don't think there is any other way to inject
a custom ApplicationMasterProtocol. But I did add a {{@VisibleForTesting}} as per [~kasha]'s

bq. The ordering of where ExecutionType goes is not consistent, in ResourceRequest is at the
end, while in most other places is earlier. Strong typing make this not a big deal, just looks
cleaner if consistent.
The reason I had to put ExecutionType in the end for {{ContainerRequest}} and {{ResourceRequest}}
was that otherwise, would have to change many of the methods marked {{@Stable}} in previous
releases.. In the AMRMClientImpl, it appears before the {{Resource}} in the composite key
for reasons i mentioend [here|https://issues.apache.org/jira/browse/YARN-5124?focusedCommentId=15303099&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15303099]
but thats an implementation detail and hidden from the user.

Hope this made sense..

> Modify AMRMClient to set the ExecutionType in the ResourceRequest
> -----------------------------------------------------------------
>                 Key: YARN-5124
>                 URL: https://issues.apache.org/jira/browse/YARN-5124
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Arun Suresh
>            Assignee: Arun Suresh
>         Attachments: YARN-5124.001.patch, YARN-5124.002.patch, YARN-5124.003.patch, YARN-5124.004.patch,
YARN-5124.005.patch, YARN-5124.006.patch, YARN-5124.008.patch, YARN-5124.009.patch, YARN-5124.010.patch,
YARN-5124_YARN-5180_combined.007.patch, YARN-5124_YARN-5180_combined.008.patch
> Currently the {{ContainerRequest}} allows the AM to set the {{ExecutionType}} in the
AMRMClient, but it is not being set in the actual {{ResourceRequest}} that is sent to the

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message