hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "zhihai xu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-6143) add configuration for mapreduce speculative execution in MR2
Date Mon, 19 Jan 2015 06:30:35 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

zhihai xu updated MAPREDUCE-6143:
---------------------------------
    Description: 
add configuration for  mapreduce speculative execution in MR2.
Currently mapreduce.job.speculative.speculativecap and mapreduce.job.speculative.slownodethreshold
are not used for MR2 mapreduce speculative execution any more.

We should make the following hardcode constants in DefaultSpeculator configurable for MR2
Map Reduce speculative execution:

private static final long SOONEST_RETRY_AFTER_NO_SPECULATE = 1000L * 1L;
private static final long SOONEST_RETRY_AFTER_SPECULATE = 1000L * 15L;
private static final double PROPORTION_RUNNING_TASKS_SPECULATABLE = 0.1;
private static final double PROPORTION_TOTAL_TASKS_SPECULATABLE = 0.01;
private static final int MINIMUM_ALLOWED_SPECULATIVE_TASKS = 10;

  was:
add configuration for  mapreduce speculative execution in MR2.
Currently mapreduce.job.speculative.speculativecap, mapreduce.job.speculative.slowtaskthreshold
and mapreduce.job.speculative.slownodethreshold are not used for MR2 mapreduce speculative
execution any more.

We should make the following hardcode constants in DefaultSpeculator configurable for MR2
Map Reduce speculative execution:

private static final long SOONEST_RETRY_AFTER_NO_SPECULATE = 1000L * 1L;
private static final long SOONEST_RETRY_AFTER_SPECULATE = 1000L * 15L;
private static final double PROPORTION_RUNNING_TASKS_SPECULATABLE = 0.1;
private static final double PROPORTION_TOTAL_TASKS_SPECULATABLE = 0.01;
private static final int MINIMUM_ALLOWED_SPECULATIVE_TASKS = 10;


> add configuration for  mapreduce speculative execution in MR2
> -------------------------------------------------------------
>
>                 Key: MAPREDUCE-6143
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6143
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv2
>    Affects Versions: 2.5.1
>            Reporter: zhihai xu
>            Assignee: zhihai xu
>
> add configuration for  mapreduce speculative execution in MR2.
> Currently mapreduce.job.speculative.speculativecap and mapreduce.job.speculative.slownodethreshold
are not used for MR2 mapreduce speculative execution any more.
> We should make the following hardcode constants in DefaultSpeculator configurable for
MR2 Map Reduce speculative execution:
> private static final long SOONEST_RETRY_AFTER_NO_SPECULATE = 1000L * 1L;
> private static final long SOONEST_RETRY_AFTER_SPECULATE = 1000L * 15L;
> private static final double PROPORTION_RUNNING_TASKS_SPECULATABLE = 0.1;
> private static final double PROPORTION_TOTAL_TASKS_SPECULATABLE = 0.01;
> private static final int MINIMUM_ALLOWED_SPECULATIVE_TASKS = 10;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message