Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: German Florez-Larrahondo <german.fl@samsung.com>
To: user@hadoop.apache.org
References: <51E81FBA.5070408@cse.ohio-state.edu>
 <CAOcnVr0FRymy2z8x8d4UjAM-QZwYH4U_ExT3GG6XV02RS7pjGg@mail.gmail.com>
In-reply-to: 
 <CAOcnVr0FRymy2z8x8d4UjAM-QZwYH4U_ExT3GG6XV02RS7pjGg@mail.gmail.com>
Subject: RE: Fault tolerance and Speculative Execution
Date: Thu, 18 Jul 2013 12:37:38 -0500
Message-id: <00a801ce83dd$7fd416a0$7f7c43e0$@samsung.com>
MIME-version: 1.0
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7bit
Thread-index: AQJsL62ipBCvPdmM4ISgBU2FOR9NYgFT5iSxmCS2S4A=
Content-language: en-us

Also, a  simple explanation of how speculative execution works and what are
the key settings  can be found here:
http://books.google.com/books?id=Wu_xeGdU4G8C&pg=PA216&dq=hadoop+the+definit
ive+guide+speculative+execution&hl=en&sa=X&ei=lyLoUd7bDojk9gTC1YDIBA&ved=0CD
wQ6AEwAA

In addition, there used to be other parameters (slownodethreshold,
slowtaskthreshold & speculativecap)
http://coffee2idea.blogspot.com/2011/11/hadoop-speculative-execution.html
but I believe they were deprecated...
 
Regards
German
./g


-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com] 
Sent: Thursday, July 18, 2013 12:11 PM
To: <user@hadoop.apache.org>
Subject: Re: Fault tolerance and Speculative Execution

What you describe in the first paragraph is not true.

Speculative execution API toggles are listed in the documentation:
http://hadoop.apache.org/docs/stable/mapred_tutorial.html#Job+Configuration
and in the mapred-default page in property form:
http://hadoop.apache.org/docs/stable/mapred-default.html. Speculative
execution is enabled by default.

On Thu, Jul 18, 2013 at 10:32 PM, Sundeep Kambhampati
<kambhamp@cse.ohio-state.edu> wrote:
> Hi all,
> Is it true that Hadoop 'always' starts same map tasks multiple times 
> in order to be fault tolerant. i.e. same task is launched on several 
> machines so that even if a node fails then same task would be 
> available on other node. And in case no node fails redundant task that
finishes late is killed.
> If it is true how can I change that configuration for Hadoop to do it 
> or not do it.
>
> Speculative execution on the other hand does what I explained above 
> (redundant map tasks) but only after all the map tasks are scheduled 
> and if some nodes are free it starts redundant map tasks for those 
> which are running slow. Is it always true? How do change this 
> configuration enable/disable.
>
> I am using Hadoop-1.1.2 incase version matters.
>
> I really appreciate if someone could help me with this. Thank you.
>
> Regards
> Sundeep
>
>


--
Harsh J