Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of yhemanth@gmail.com designates
 209.85.160.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=XSHeC5gsdYjn0kFp2E9sJJZ35A2o5778+2HFUyrOuSzvqLuR7zcdvAZEHgjAsUKBd9
         DjX4fz/wPJJ4KSC5XffLtju5kUJPTEjV0uR4KQVExvJzx5TYPUw5/7DgtcWSXyzG/LAn
         KnmyN+KEqsRAbqXGwkcL+r8A60LjU3W78JFzU=
MIME-Version: 1.0
In-Reply-To: <AANLkTik4Fmg0kGCKzarYKqdWkXky8LcJWfs8hdQEg7fg@mail.gmail.com>
References: <AANLkTik4Fmg0kGCKzarYKqdWkXky8LcJWfs8hdQEg7fg@mail.gmail.com>
Date: Mon, 26 Jul 2010 09:51:58 +0530
Message-ID: <AANLkTikRYmveZsTYkBSN6XDd1aQubgmTg6YcN5z6VO-F@mail.gmail.com>
Subject: Re: What's speculative tasks
From: Hemanth Yamijala <yhemanth@gmail.com>
To: mapreduce-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Pedro,

On Sun, Jul 25, 2010 at 7:33 PM, Pedro Costa <psdc1978@gmail.com> wrote:
> Hi,
>
> - In hadoop MR it's used the term speculative tasks. What is speculative
> tasks?
>

When the MR framework detects that some tasks are running slower than
others in the job, it has an option to launch duplicates of those
tasks on different nodes from the original ones, with the hope that
they would complete sooner than the original slow tasks. The
motivation for this feature is that it has been found that every job
has 'stragglers' - a small percentage of tasks that are significantly
slower than the rest of them and these slow down the overall execution
time of the job. Typically these stragglers come around due to bad
hardware.

> - During the execution of a MR test, when we don't have splits to attribute
> to reduce tasks, those reduce tasks will run? For example, if I set that
> will run 6 reduce tasks and I don't have splits during the running of the
> example, the reduce tasks will run? If so, where is verified that a reduce
> task has a split assigned?

Splits are related to map tasks, not reduce tasks. Reduce tasks get
their input from the output of map tasks that is generated and stored
in an intermediate fashion on the compute nodes. Can you clarify what
you are looking for, with this context ?

Thanks
Hemanth