hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Kanter (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-6871) Allow users to specify racks and nodes for strict locality for AMs
Date Thu, 20 Apr 2017 00:30:04 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Robert Kanter updated MAPREDUCE-6871:
    Attachment: MAPREDUCE-6871.004.patch

The 004 patch:
- Renames the config to {{mapreduce.job.am.strict-locality}}
- Adds a debug logging check
- Removes the redundant label setting code in a test (I think that's a holdover from an earlier

[~kasha], for your feedback about the {{rackRequests}}: it does need to be a map.  If someone
specifies {{/rack1,/rack1/node1}}, we'd need to update the already created request for rack1
to have the strict locality set to false.

> Allow users to specify racks and nodes for strict locality for AMs
> ------------------------------------------------------------------
>                 Key: MAPREDUCE-6871
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6871
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: client
>            Reporter: Robert Kanter
>            Assignee: Robert Kanter
>         Attachments: MAPREDUCE-6871.001.patch, MAPREDUCE-6871.002.patch, MAPREDUCE-6871.003.patch,
> YARN-6050 fixed the YARN API to allow multiple {{ResourceRequest}}'s when submitting
an AM so that you can actually do rack or node locality.  We should allow MapReduce users
to take advantage of this by exposing this functionality in some way.  The raw YARN API allows
for a lot of flexibility (e.g. different resources per request, etc), but we don't necessarily
want to allow the user to do too much here so they don't shoot themselves in the foot and
we don't make this overly complicated.  
> I propose we allow users to specify racks and nodes for strict locality.  This would
allow users to restrict an MR AM to specific racks and/or nodes.  We could add a new property,
{{mapreduce.job.am.resource-request.strict.locality}}, which takes a comma-separated list
of entries like:
> - {{/<rack>}}
> - {{/<rack>/<node>}}
> - {{<node>}} (assumes /default-rack)
> MapReduce would then use this information to create the corresponding {{ResourceRequest}}'s.
> For example, {{mapreduce.job.am.resource-request.strict.locality=/rack1/node1}} would
create the following {{ResourceRequest}}'s:
> - resourceName=ANY, relaxLocality=false, capability=<X,Y>
> - resourceName=/rack1, relaxLocality=false, capability=<X,Y>
> - resourceName=node1, relaxLocality=true, capability=<X,Y>
> By default, the property would be unset, and you'd get the normal {{ANY}} {{ResourceRequest}}.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message