hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-2214) Implement a Map-Reduce application which can be used to reliably launch speculative tasks
Date Fri, 16 Nov 2007 07:37:43 GMT
Implement a Map-Reduce application which can be used to reliably launch speculative tasks
-----------------------------------------------------------------------------------------

                 Key: HADOOP-2214
                 URL: https://issues.apache.org/jira/browse/HADOOP-2214
             Project: Hadoop
          Issue Type: Test
          Components: test
            Reporter: Arun C Murthy
             Fix For: 0.16.0


It would be very useful to have a reliable test case to help launch speculative tasks (maps
and/or reduces) to help debug problems or fine-tune strategies for speculative execution.

I propose we implement some along the lines of the SmallJobsBenchmark and put it in the hadoop-test.jar.
I imagine it could be used like:

{noformat}
$ bin/hadoop jar hadoop-${version}-test.jar speculativejob -nMaps 1000 -nReduces 50 -specMaps
10 -specReduces 5
{noformat}

The application should be minutely aware of the _current_ strategies for launching speculative
tasks, and throttle requisite no. of maps/reduces to ensure that the speculative tasks are
launched in a reliable and repeatable manner.

Thoughts?


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message