Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8C5CB77BA for ; Thu, 22 Sep 2011 20:26:50 +0000 (UTC) Received: (qmail 1284 invoked by uid 500); 22 Sep 2011 20:26:49 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 1248 invoked by uid 500); 22 Sep 2011 20:26:49 -0000 Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-user@hadoop.apache.org Delivered-To: mailing list mapreduce-user@hadoop.apache.org Received: (qmail 1240 invoked by uid 99); 22 Sep 2011 20:26:49 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Sep 2011 20:26:49 +0000 X-ASF-Spam-Status: No, hits=-2.8 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_HI,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of matthew.goeke@monsanto.com designates 164.144.240.26 as permitted sender) Received: from [164.144.240.26] (HELO gateway1.monsanto.com) (164.144.240.26) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Sep 2011 20:26:42 +0000 X-IronPort-AV: E=Sophos;i="4.68,425,1312174800"; d="scan'208,217";a="52827732" Received: from unknown (HELO NA1000EXR01.na.ds.monsanto.com) ([10.29.223.249]) by gateway1.monsanto.com with ESMTP; 22 Sep 2011 15:27:46 -0500 Received: from NA1000EXR01.na.ds.monsanto.com ([10.30.64.43]) by NA1000EXR01.na.ds.monsanto.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 22 Sep 2011 15:26:21 -0500 Received: from stlwexhubprd02.na.ds.monsanto.com ([10.30.51.35]) by NA1000EXR01.na.ds.monsanto.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 22 Sep 2011 15:26:20 -0500 Received: from stlwexmbxprd04.na.ds.monsanto.com ([169.254.7.20]) by stlwexhubprd02.na.ds.monsanto.com ([10.30.51.35]) with mapi id 14.01.0255.000; Thu, 22 Sep 2011 15:26:20 -0500 From: "GOEKE, MATTHEW (AG/1000)" To: "mapreduce-user@hadoop.apache.org" Subject: RE: FairScheduler Local Task Restriction Thread-Topic: FairScheduler Local Task Restriction Thread-Index: Acx5X5ob1fBT+aD6RamvrXKZGXJqewAAjY4AAAB1SWAAAFsVcAAAHqUA Date: Thu, 22 Sep 2011 20:26:18 +0000 Message-ID: References: <2D3A1C35D7BA764A89D1B6166D213AB04FC7BC0545@TINY.corp.clearedgeit.com> <2D3A1C35D7BA764A89D1B6166D213AB04FC7BC054B@TINY.corp.clearedgeit.com> <2D3A1C35D7BA764A89D1B6166D213AB04FC7BC054D@TINY.corp.clearedgeit.com> In-Reply-To: <2D3A1C35D7BA764A89D1B6166D213AB04FC7BC054D@TINY.corp.clearedgeit.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.30.3.246] Content-Type: multipart/alternative; boundary="_000_E236CD55EC618B4884CA7273B31AA0780753F738stlwexmbxprd04n_" MIME-Version: 1.0 X-OriginalArrivalTime: 22 Sep 2011 20:26:20.0992 (UTC) FILETIME=[E3F31C00:01CC7965] X-Virus-Checked: Checked by ClamAV on apache.org --_000_E236CD55EC618B4884CA7273B31AA0780753F738stlwexmbxprd04n_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Just to confirm your configuration, how many logical cores do these boxes a= ctually have (I am assuming dual quad core HT'ed)? Do you not have any redu= ce slots allocated? Matt From: Adam Shook [mailto:ashook@clearedgeit.com] Sent: Thursday, September 22, 2011 3:22 PM To: mapreduce-user@hadoop.apache.org Subject: RE: FairScheduler Local Task Restriction Okay, I put a Thread.sleep to test my theory and it will run all 128 at a t= ime - they are just completing too quickly. I guess there is no other way = to get around it, unless someone knows how to make the scheduler schedule f= aster... -- Adam From: Adam Shook [mailto:ashook@clearedgeit.com] Sent: Thursday, September 22, 2011 4:17 PM To: mapreduce-user@hadoop.apache.org Subject: RE: FairScheduler Local Task Restriction It's an 8 node cluster and all 8 task trackers are being used. Each tracke= r has 16 max map tasks. Each tracker seems to be running two at a time. M= ap tasks take 10 seconds from start to finish. Is it possible that they ar= e just completing faster than they can be created and it just seems to stic= k around 16? -- Adam From: GOEKE, MATTHEW (AG/1000) [mailto:matthew.goeke@monsanto.com] Sent: Thursday, September 22, 2011 4:06 PM To: mapreduce-user@hadoop.apache.org Subject: RE: FairScheduler Local Task Restriction If you dig into the job history on the web-ui can you confirm whether it is= the same 16 tasktrackers slots that are getting the map tasks? Long shot b= ut it could be that it is actually distributing across your cluster and the= re is some other issue that is springing up. Also, how long does each of yo= ur map tasks take? Matt From: Adam Shook [mailto:ashook@clearedgeit.com] Sent: Thursday, September 22, 2011 2:41 PM To: mapreduce-user@hadoop.apache.org Subject: FairScheduler Local Task Restriction Hello All, I have recently switched my small Hadoop dev cluster (v0.20.1) to use the F= airScheduler. I have a max of 128 map tasks available and recently noticed= that my jobs seem to use a maximum of 16 at any given time (the job I am l= ooking at in particular runs for about 15 minutes) - they are also all data= local map tasks. I searched around a bit and discovered the mapred.fairsc= heduler.locality.delay may be to blame. I set it to 0 in mapred-site.xml, = copied the file around to my nodes and tried running another job. It still= has 16 tasks. Does it require a cluster restart? Is it something totally different? Sho= uld I not set this value to zero? Thanks! -- Adam This e-mail message may contain privileged and/or confidential information,= and is intended to be received only by persons entitled to receive such information. If you have received this e-mail in error, ple= ase notify the sender immediately. Please delete it and all attachments from any servers, hard drives or any other media. Other use= of this e-mail by you is strictly prohibited. All e-mails and attachments sent and received are subject to monitoring, re= ading and archival by Monsanto, including its subsidiaries. The recipient of this e-mail is solely responsible for checki= ng for the presence of "Viruses" or other "Malware". Monsanto, along with its subsidiaries, accepts no liability for any damage = caused by any such code transmitted by or accompanying this e-mail or any attachment. The information contained in this email may be subject to the export contro= l laws and regulations of the United States, potentially including but not limited to the Export Administration Regulations (EAR) an= d sanctions regulations issued by the U.S. Department of Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this = information you are obligated to comply with all applicable U.S. export laws and regulations. ________________________________ No virus found in this message. Checked by AVG - www.avg.com Version: 10.0.1410 / Virus Database: 1520/3912 - Release Date: 09/22/11 ________________________________ No virus found in this message. Checked by AVG - www.avg.com Version: 10.0.1410 / Virus Database: 1520/3912 - Release Date: 09/22/11 --_000_E236CD55EC618B4884CA7273B31AA0780753F738stlwexmbxprd04n_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Just to confirm your c= onfiguration, how many logical cores do these boxes actually have (I am ass= uming dual quad core HT’ed)? Do you not have any reduce slots allocat= ed?

 

Matt=

 

From: Adam Sho= ok [mailto:ashook@clearedgeit.com]
Sent: Thursday, September 22, 2011 3:22 PM
To: mapreduce-user@hadoop.apache.org
Subject: RE: FairScheduler Local Task Restriction<= /p>

 

Okay, I put a Thread.s= leep to test my theory and it will run all 128 at a time – they are j= ust completing too quickly.  I guess there is no other way to get arou= nd it, unless someone knows how to make the scheduler schedule faster...

 

-- Adam

 

From: Adam Sho= ok [mailto:ashook@clearedgeit.com]
Sent: Thursday, September 22, 2011 4:17 PM
To: mapreduce-user@hadoop.apache.org
Subject: RE: FairScheduler Local Task Restriction<= /p>

 

It’s an 8 node c= luster and all 8 task trackers are being used.  Each tracker has 16 ma= x map tasks.  Each tracker seems to be running two at a time.  Ma= p tasks take 10 seconds from start to finish.  Is it possible that they are just completing faster than they can be created and it just = seems to stick around 16?

 

-- Adam

 

From: GOEKE, M= ATTHEW (AG/1000) [mailto:matthew.goeke@monsanto.com]
Sent: Thursday, September 22, 2011 4:06 PM
To: mapreduce-user@hadoop.apache.org
Subject: RE: FairScheduler Local Task Restriction<= /p>

 

If you dig into the jo= b history on the web-ui can you confirm whether it is the same 16 tasktrack= ers slots that are getting the map tasks? Long shot but it could be that it= is actually distributing across your cluster and there is some other issue that is springing up. Also, how long= does each of your map tasks take?

 

Matt=

 

From: Adam Sho= ok [mailto:ashook@clearedgeit.com]
Sent: Thursday, September 22, 2011 2:41 PM
To: mapreduce-user@hadoop.apache.org
Subject: FairScheduler Local Task Restriction

 

Hello All,

 

I have recently switched my small Hadoop dev cluster= (v0.20.1) to use the FairScheduler.  I have a max of 128 map tasks av= ailable and recently noticed that my jobs seem to use a maximum of 16 at an= y given time (the job I am looking at in particular runs for about 15 minutes) – they are also all data local= map tasks.  I searched around a bit and discovered the mapred.fairsch= eduler.locality.delay may be to blame.  I set it to 0 in mapred-site.x= ml, copied the file around to my nodes and tried running another job.  It still has 16 tasks.

 

Does it require a cluster restart?  Is it somet= hing totally different?  Should I not set this value to zero?

 

Thanks!

 

-- Adam

This e-mail message may contain privileged and/or confiden= tial information, and is intended to be received only by persons entitled to receive such information. If you have received this e-mail in error, ple= ase notify the sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use= of this e-mail by you is strictly prohibited.

All e-mails and attachments sent and received are subject to monitoring, re= ading and archival by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for checki= ng for the presence of "Viruses" or other "Malware". Monsanto, along with its subsidiaries, accepts no liability for any damage = caused by any such code transmitted by or accompanying
this e-mail or any attachment.


The information contained in this email may be subject to the export contro= l laws and regulations of the United States, potentially
including but not limited to the Export Administration Regulations (EAR) an= d sanctions regulations issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC).  As a recipient of = this information you are obligated to comply with all
applicable U.S. export laws and regulations.

No virus found in this message.
Checked by AVG - www.avg.com
Version: 10.0.1410 / Virus Database: 1520/3912 - Release Date: 09/22/11

No virus found in this message.
Checked by AVG - www.avg.com
Version: 10.0.1410 / Virus Database: 1520/3912 - Release Date: 09/22/11

--_000_E236CD55EC618B4884CA7273B31AA0780753F738stlwexmbxprd04n_--