Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Date: Wed, 22 Feb 2012 19:05:49 +0000 (UTC)
From: "Nathan Roberts (Commented) (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: 
 <1140310909.5478.1329937549094.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <909173124.4658.1329926989222.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (MAPREDUCE-3895) Speculative execution algorithm
 in 1.0 is too pessimistic in many cases
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-3895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13213862#comment-13213862 ] 

Nathan Roberts commented on MAPREDUCE-3895:
-------------------------------------------

If the 1 minute or the 20% were configurable, I think that could capture a large amount of the use cases without the risk of instability. Not nearly an ideal solution but maybe enough to avoid the large overhead currently being consumed??
                
> Speculative execution algorithm in 1.0 is too pessimistic in many cases
> -----------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3895
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3895
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker, performance
>    Affects Versions: 1.0.0
>            Reporter: Nathan Roberts
>
> We are seeing many instances where largish jobs are ending up with 30-50% of reduce tasks being speculatively re-executed. This can be a significant drain on cluster resources. 
> The primary reason is due to the way progress in the reduce phase can make huge jumps in a very short amount of time. This fact leads the speculative execution code to think lots of tasks have fallen way behind the average when in fact they haven't
> The important piece of the algorithm is essentially:
> * Am I more than 20% behind the average progress?
> * Have I been running for at least a minute?
> * Have any tasks completed yet?
> Unfortunately, a set of reduce tasks which spend a couple of minutes in the Copy phase, and very little time in the Sort phase, will trigger all these conditions for a large percentage of the reduce tasks. (the tasks' progress jump from 33% to 66% almost instantly which then triggers the speculation). I've seen this on several very large jobs which spend about 2 minutes in Copy, a few seconds in Sort, and 40 minutes in Reduce. These jobs launch about 30-40% additional reduce tasks which then run for almost the full 40 minutes. 
> This area becomes more plugable in MRv2 but for 1.0 it would be good if some portion of this algorithm could be configurable so that a job could have some degree of control (just disabling speculative execution is not really an option). 
>  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira