Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
Message-ID: <16495008.1195198663230.JavaMail.jira@brutus>
Date: Thu, 15 Nov 2007 23:37:43 -0800 (PST)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Created: (HADOOP-2214) Implement a Map-Reduce application
 which can be used to reliably launch speculative tasks
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Implement a Map-Reduce application which can be used to reliably launch speculative tasks
-----------------------------------------------------------------------------------------

                 Key: HADOOP-2214
                 URL: https://issues.apache.org/jira/browse/HADOOP-2214
             Project: Hadoop
          Issue Type: Test
          Components: test
            Reporter: Arun C Murthy
             Fix For: 0.16.0


It would be very useful to have a reliable test case to help launch speculative tasks (maps and/or reduces) to help debug problems or fine-tune strategies for speculative execution.

I propose we implement some along the lines of the SmallJobsBenchmark and put it in the hadoop-test.jar. I imagine it could be used like:

{noformat}
$ bin/hadoop jar hadoop-${version}-test.jar speculativejob -nMaps 1000 -nReduces 50 -specMaps 10 -specReduces 5
{noformat}

The application should be minutely aware of the _current_ strategies for launching speculative tasks, and throttle requisite no. of maps/reduces to ensure that the speculative tasks are launched in a reliable and repeatable manner.

Thoughts?


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.