hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srinivas Chamarthi <srinivas.chamar...@gmail.com>
Subject Re: Query about Speculative Execution
Date Fri, 05 Dec 2014 10:40:34 GMT

may I know where should I find the sources related to speculative
scheduling happens ? and also how do we discard the output coming from
already completed mapper ?

I am actually trying to do something similar, like spawn map tasks
redundantly but not for speculative reasons but for each mapper and reducer
so that I can do a integrity check between the nodes where the tasks are

any help is greatly appreciated.


On Thu, Dec 6, 2012 at 8:40 PM, Ajay Srivastava <Ajay.Srivastava@guavus.com>

>  Thanks Mahesh & Harsh.
>  On 07-Dec-2012, at 7:42 AM, Mahesh Balija wrote:
> To simply, if you turn-off the speculative execution then the system will
> never bother about slow running jobs unless they won't report beyond
> specified time (10 minutes).
> If you have set speculative execution to true then the system may spawn
> another instance of mapper and consider the output of the fast running once
> or early completing task.
> Best,
> Mahesh Balija,
> Calsoft Labs.
> On Thu, Dec 6, 2012 at 8:27 PM, Ajay Srivastava <
> Ajay.Srivastava@guavus.com> wrote:
>> Hi,
>> What is the behavior of jobTracker if speculative execution is off and a
>> task on data node is running extremely slow?
>> Will the jobTracker simply wait till the slow running task finishes or it
>> will try to heal the situation? Assuming that heartbeat from the node
>> running slow task are regular.
>> Regards,
>> Ajay Srivastava

View raw message