From mapreduce-dev-return-8284-apmail-hadoop-mapreduce-dev-archive=hadoop.apache.org@hadoop.apache.org Thu Sep 13 14:39:48 2012 Return-Path: X-Original-To: apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E4D6492CE for ; Thu, 13 Sep 2012 14:39:48 +0000 (UTC) Received: (qmail 17545 invoked by uid 500); 13 Sep 2012 14:39:48 -0000 Delivered-To: apmail-hadoop-mapreduce-dev-archive@hadoop.apache.org Received: (qmail 17480 invoked by uid 500); 13 Sep 2012 14:39:48 -0000 Mailing-List: contact mapreduce-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-dev@hadoop.apache.org Delivered-To: mailing list mapreduce-dev@hadoop.apache.org Received: (qmail 17470 invoked by uid 99); 13 Sep 2012 14:39:48 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Sep 2012 14:39:48 +0000 X-ASF-Spam-Status: No, hits=1.1 required=5.0 tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: 98.139.253.104 is neither permitted nor denied by domain of evans@yahoo-inc.com) Received: from [98.139.253.104] (HELO mrout1-b.corp.bf1.yahoo.com) (98.139.253.104) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Sep 2012 14:39:40 +0000 Received: from SP1-EX07CAS04.ds.corp.yahoo.com (sp1-ex07cas04.corp.sp1.yahoo.com [216.252.116.155]) by mrout1-b.corp.bf1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id q8DEcsaO000414 for ; Thu, 13 Sep 2012 07:38:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yahoo-inc.com; s=cobra; t=1347547135; bh=lhEJWl6nvUcQ73CQ+F5So1JrQIOZoQ8fF1o8k+n26M0=; h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type: Content-Transfer-Encoding:MIME-Version; b=JmsyRr4i2y9dXNe0Bt1P/W8375MAnm9/FmuV6f7EtrxEsKNwLn9cvFHVSsKIePZTe OI2OR3AnpTGG1T6Eo37F0Gu8ak6R2t/nQWY/4FssZDM59uzSc3npwA3e9JpN/ZFMjl 8GgGsacEuA5o7R2x0VVbhAGJI/FGp7eWBSvflpRk= Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by SP1-EX07CAS04.ds.corp.yahoo.com ([216.252.116.158]) with mapi; Thu, 13 Sep 2012 07:38:54 -0700 From: Robert Evans To: "mapreduce-dev@hadoop.apache.org" Date: Thu, 13 Sep 2012 07:38:50 -0700 Subject: Re: Speculative Execution... Thread-Topic: Speculative Execution... Thread-Index: Ac2RvX+tIQycLABAQsq3lsO6DyTt4Q== Message-ID: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.2.3.120616 acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Milter-Version: master.31+4-gbc07cd5+ X-CLX-ID: 547135002 Under YARN (branch-2, branch-0.23, and trunk) the speculative execution decision is pluggable, and can be replaced by a user. If you could come up with a better solution to speculative execution that would be great. We have known for a while that it is not very good (most of the time we run a speculative task it is just wasted). In branch-2 we have a new version that we think is better (more conservative), but I am not sure how much of a study has been done on exactly how much better it is or what else can be done to make it even better. I would look at adding your ideas into that plugin and not so much of using the config to turn speculation on or off dynamically, because there are some map/reduce applications that abuse map/reduce somewhat and will not run correctly if speculative execution is enabled. --Bobby Evans=20 On 9/13/12 1:08 AM, "Suresh S" wrote: >Hello Sir/Madam.... > > I am doing PhD. I am interested to do research in Hadoop for publishing >paper. >I know little bit about speculative execution of slow tasks. I know it is >possible to enable or disable speculative execution. > >But, > Is there any idea published already for dynamically enable or disable >the speculative execution depending on application, cluster load and other >run time parameters? > > Is it worth to do research in this direction? > > Is this contribution is worth to publish a conference or journal >paper(s)? > >*Regards* >*S.Suresh,* >*Research Scholar,* >*Department of Computer Applications,* >*National Institute of Technology,* >*Tiruchirappalli - 620015.* >*India* >*Mobile: +91-9941506562*