Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 704A173D3 for ; Thu, 22 Sep 2011 20:06:30 +0000 (UTC) Received: (qmail 38837 invoked by uid 500); 22 Sep 2011 20:06:29 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 38790 invoked by uid 500); 22 Sep 2011 20:06:29 -0000 Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-user@hadoop.apache.org Delivered-To: mailing list mapreduce-user@hadoop.apache.org Received: (qmail 38782 invoked by uid 99); 22 Sep 2011 20:06:29 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Sep 2011 20:06:29 +0000 X-ASF-Spam-Status: No, hits=-2.8 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_HI,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of matthew.goeke@monsanto.com designates 164.144.240.27 as permitted sender) Received: from [164.144.240.27] (HELO gateway2.monsanto.com) (164.144.240.27) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Sep 2011 20:06:21 +0000 X-IronPort-AV: E=Sophos;i="4.68,425,1312174800"; d="scan'208,217";a="51056468" Received: from unknown (HELO NA1000EXR02.na.ds.monsanto.com) ([10.29.223.250]) by gateway2.monsanto.com with ESMTP; 22 Sep 2011 15:07:24 -0500 Received: from NA1000EXR02.na.ds.monsanto.com ([10.30.64.42]) by NA1000EXR02.na.ds.monsanto.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 22 Sep 2011 15:05:59 -0500 Received: from stlwexhubprd03.na.ds.monsanto.com ([10.30.58.187]) by NA1000EXR02.na.ds.monsanto.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 22 Sep 2011 15:05:59 -0500 Received: from stlwexchhubp01.na.ds.monsanto.com (10.30.58.178) by stlwexhubprd03.na.ds.monsanto.com (10.30.58.187) with Microsoft SMTP Server (TLS) id 14.1.255.0; Thu, 22 Sep 2011 15:05:58 -0500 Received: from stlwexmbxprd04.na.ds.monsanto.com ([169.254.7.20]) by stlwexchhubp01.na.ds.monsanto.com ([10.30.58.178]) with mapi id 14.01.0255.000; Thu, 22 Sep 2011 15:05:57 -0500 From: "GOEKE, MATTHEW (AG/1000)" To: "mapreduce-user@hadoop.apache.org" Subject: RE: FairScheduler Local Task Restriction Thread-Topic: FairScheduler Local Task Restriction Thread-Index: Acx5X5ob1fBT+aD6RamvrXKZGXJqewAAjY4A Date: Thu, 22 Sep 2011 20:05:57 +0000 Message-ID: References: <2D3A1C35D7BA764A89D1B6166D213AB04FC7BC0545@TINY.corp.clearedgeit.com> In-Reply-To: <2D3A1C35D7BA764A89D1B6166D213AB04FC7BC0545@TINY.corp.clearedgeit.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.30.3.246] Content-Type: multipart/alternative; boundary="_000_E236CD55EC618B4884CA7273B31AA0780753F68Estlwexmbxprd04n_" MIME-Version: 1.0 X-OriginalArrivalTime: 22 Sep 2011 20:05:59.0495 (UTC) FILETIME=[0BE17570:01CC7963] X-Virus-Checked: Checked by ClamAV on apache.org --_000_E236CD55EC618B4884CA7273B31AA0780753F68Estlwexmbxprd04n_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable If you dig into the job history on the web-ui can you confirm whether it is= the same 16 tasktrackers slots that are getting the map tasks? Long shot b= ut it could be that it is actually distributing across your cluster and the= re is some other issue that is springing up. Also, how long does each of yo= ur map tasks take? Matt From: Adam Shook [mailto:ashook@clearedgeit.com] Sent: Thursday, September 22, 2011 2:41 PM To: mapreduce-user@hadoop.apache.org Subject: FairScheduler Local Task Restriction Hello All, I have recently switched my small Hadoop dev cluster (v0.20.1) to use the F= airScheduler. I have a max of 128 map tasks available and recently noticed= that my jobs seem to use a maximum of 16 at any given time (the job I am l= ooking at in particular runs for about 15 minutes) - they are also all data= local map tasks. I searched around a bit and discovered the mapred.fairsc= heduler.locality.delay may be to blame. I set it to 0 in mapred-site.xml, = copied the file around to my nodes and tried running another job. It still= has 16 tasks. Does it require a cluster restart? Is it something totally different? Sho= uld I not set this value to zero? Thanks! -- Adam This e-mail message may contain privileged and/or confidential information,= and is intended to be received only by persons entitled to receive such information. If you have received this e-mail in error, ple= ase notify the sender immediately. Please delete it and all attachments from any servers, hard drives or any other media. Other use= of this e-mail by you is strictly prohibited. All e-mails and attachments sent and received are subject to monitoring, re= ading and archival by Monsanto, including its subsidiaries. The recipient of this e-mail is solely responsible for checki= ng for the presence of "Viruses" or other "Malware". Monsanto, along with its subsidiaries, accepts no liability for any damage = caused by any such code transmitted by or accompanying this e-mail or any attachment. The information contained in this email may be subject to the export contro= l laws and regulations of the United States, potentially including but not limited to the Export Administration Regulations (EAR) an= d sanctions regulations issued by the U.S. Department of Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this = information you are obligated to comply with all applicable U.S. export laws and regulations. --_000_E236CD55EC618B4884CA7273B31AA0780753F68Estlwexmbxprd04n_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

If you dig into the jo= b history on the web-ui can you confirm whether it is the same 16 tasktrack= ers slots that are getting the map tasks? Long shot but it could be that it= is actually distributing across your cluster and there is some other issue that is springing up. Also, how long= does each of your map tasks take?

 

Matt=

 

From: Adam Sho= ok [mailto:ashook@clearedgeit.com]
Sent: Thursday, September 22, 2011 2:41 PM
To: mapreduce-user@hadoop.apache.org
Subject: FairScheduler Local Task Restriction

 

Hello All,

 

I have recently switched my small Hadoop dev cluster= (v0.20.1) to use the FairScheduler.  I have a max of 128 map tasks av= ailable and recently noticed that my jobs seem to use a maximum of 16 at an= y given time (the job I am looking at in particular runs for about 15 minutes) – they are also all data local= map tasks.  I searched around a bit and discovered the mapred.fairsch= eduler.locality.delay may be to blame.  I set it to 0 in mapred-site.x= ml, copied the file around to my nodes and tried running another job.  It still has 16 tasks.

 

Does it require a cluster restart?  Is it somet= hing totally different?  Should I not set this value to zero?

 

Thanks!

 

-- Adam

This e-mail message may contain privileged and/or = confidential information, and is intended to be received only by persons en= titled
to receive such information. If you have received this e-mail in error, ple= ase notify the sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use= of this e-mail by you is strictly prohibited.

All e-mails and attachments sent and received are subject to monitoring, re= ading and archival by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for checki= ng for the presence of "Viruses" or other "Malware".
Monsanto, along with its subsidiaries, accepts no liability for any damage = caused by any such code transmitted by or accompanying
this e-mail or any attachment.


The information contained in this email may be subject to the export contro= l laws and regulations of the United States, potentially
including but not limited to the Export Administration Regulations (EAR) an= d sanctions regulations issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC).  As a recipient of = this information you are obligated to comply with all
applicable U.S. export laws and regulations. --_000_E236CD55EC618B4884CA7273B31AA0780753F68Estlwexmbxprd04n_--