hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-6703) Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers
Date Wed, 18 May 2016 13:31:13 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Arun Suresh updated MAPREDUCE-6703:
    Attachment: MAPREDUCE-6703.001.patch

Uploading initial patch

* Changes to allow the MRAppMaster to request a mix of *OPPORTUNISTIC* and *GUARANTEED* containers
for it first attempt map tasks (reduce tasks and maps to be retried are always GUARANTEED)
* User specifies number of *OPPORTUNISTIC* Containers via the *mapreduce.job.num-opportunistic-maps-per-100*
job conf : Default value is 0, which implies all maps will be guaranteed. A value of 100 means
all maps will be requested as opportunistic. For any other value say _x_, the *FIRST* _x_
maps requested by the AM will be opportunistic. If the total number of maps for the job <
_x_, then *ALL* maps will be opportunistic
* The patch also contains a test case {{TestMROpportunisticMaps}} that uses the *MiniYARNCluster*
to run a simple external sorting MR job using the above flag.

*PS* : This feature currently requires Distributed Scheduling (YARN-2885), NM Queuing (YARN-2883)
and the AMRMProxyService (YARN-2884) to be turned on in the yarn cluster.

> Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers
> ----------------------------------------------------------------------
>                 Key: MAPREDUCE-6703
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6703
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Arun Suresh
>            Assignee: Arun Suresh
>         Attachments: MAPREDUCE-6703.001.patch
> YARN-2882 and YARN-4335 introduces the concept of container ExecutionTypes and specifically
> The default ExecutionType is GUARANTEED. This JIRA proposes to allow users to provide
hints via config to the MR framework as to the number of containers it would like to schedule

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message