Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C858510861 for ; Wed, 29 Jan 2014 06:55:38 +0000 (UTC) Received: (qmail 4184 invoked by uid 500); 29 Jan 2014 06:55:29 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 3501 invoked by uid 500); 29 Jan 2014 06:55:25 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 3488 invoked by uid 99); 29 Jan 2014 06:55:24 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jan 2014 06:55:24 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_IMAGE_ONLY_28,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of amrnablus@gmail.com designates 209.85.215.49 as permitted sender) Received: from [209.85.215.49] (HELO mail-la0-f49.google.com) (209.85.215.49) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jan 2014 06:55:19 +0000 Received: by mail-la0-f49.google.com with SMTP id y1so1158384lam.36 for ; Tue, 28 Jan 2014 22:54:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=07w2WmwlP9/qLiCugDFrDFhuljeNetQLEW4kXOceLUE=; b=uGcFggVxK/nLzlHzCM6M09PEbPTI3whX07g2cAzTiHTu/eHNJS0uKn1gFAC2hC8YHh 7ujjx0MGa8Lf/ncF3Monn/2vi1T7xkpAPeCwtSzIPMls88UrND/4XWKz53rmLk03E/CU AxV3Nvjcr12W/pRIu+WFQ3oVcp6kQHcSwjTJka4tWWHhmlFhJGCd2922fZGVyZBALWl+ 1r2AlFlVq0dAwQd2vJ3jRX/1q6Hr+P8fdB0cMjo2G4jjvIL/IhVvWBlgGGcLPaE4NAFt aDYMG4qDAXJNoo4W/mSsVaTUbDjAr/qfNAxIY6oQnAx/B33zyTlsbIUg3tyqWp9nytCa mMNA== X-Received: by 10.152.220.170 with SMTP id px10mr4238602lac.13.1390978497692; Tue, 28 Jan 2014 22:54:57 -0800 (PST) MIME-Version: 1.0 Received: by 10.114.200.67 with HTTP; Tue, 28 Jan 2014 22:54:37 -0800 (PST) In-Reply-To: <9EACF911-AFAA-409D-9411-7F9DB2591F9F@keithwiley.com> References: <280A85F8-DBB1-483D-843D-0D99C3356689@keithwiley.com> <9EACF911-AFAA-409D-9411-7F9DB2591F9F@keithwiley.com> From: Amr Shahin X-Goomoji-Body: true Date: Wed, 29 Jan 2014 10:54:37 +0400 Message-ID: Subject: Re: Force one mapper per machine (not core)? To: user Content-Type: multipart/related; boundary=001a113497783ee09d04f1166bab X-Virus-Checked: Checked by ClamAV on apache.org --001a113497783ee09d04f1166bab Content-Type: multipart/alternative; boundary=001a113497783ee09a04f1166baa --001a113497783ee09a04f1166baa Content-Type: text/plain; charset=ISO-8859-1 -in theory this should work- Find the part of hadoop code that calculates the number of cores and patch it to always return one. [?] On Wed, Jan 29, 2014 at 3:41 AM, Keith Wiley wrote: > Yeah, it isn't, not even remotely, but thanks. > > On Jan 28, 2014, at 14:06 , Bryan Beaudreault wrote: > > > If this cluster is being used exclusively for this goal, you could just > set the mapred.tasktracker.map.tasks.maximum to 1. > > > > > > On Tue, Jan 28, 2014 at 5:00 PM, Keith Wiley > wrote: > > I'm running a program which in the streaming layer automatically > multithreads and does so by automatically detecting the number of cores on > the machine. I realize this model is somewhat in conflict with Hadoop, but > nonetheless, that's what I'm doing. Thus, for even resource utilization, > it would be nice to not only assign one mapper per core, but only one > mapper per machine. I realize that if I saturate the cluster none of this > really matters, but consider the following example for clarity: 4-core > nodes, 10-node cluster, thus 40 slots, fully configured across mappers and > reducers (40 slots of each). Say I run this program with just two mappers. > It would run much more efficiently (in essentially half the time) if I > could force the two mappers to go to slots on two separate machines instead > of running the risk that Hadoop may assign them both to the same machine. > > > > Can this be done? > > > > Thanks. > > > ________________________________________________________________________________ > Keith Wiley kwiley@keithwiley.com keithwiley.com > music.keithwiley.com > > "Luminous beings are we, not this crude matter." > -- Yoda > > ________________________________________________________________________________ > > --001a113497783ee09a04f1166baa Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
-in theory this should work-
Find the part of ha= doop code that calculates the number of cores and patch it to always return= one.=A0


On Wed, Jan 2= 9, 2014 at 3:41 AM, Keith Wiley <kwiley@keithwiley.com> = wrote:
Yeah, it isn't, not even remotely, but t= hanks.

On Jan 28, 2014, at 14:06 , Bryan Beaudreault wrote:

> If this cluster is being used exclusively for this goal, you could jus= t set the mapred.tasktracker.map.tasks.maximum to 1.
>
>
> On Tue, Jan 28, 2014 at 5:00 PM, Keith Wiley <kwiley@keithwiley.com> wrote:
> I'm running a program which in the streaming layer automatically m= ultithreads and does so by automatically detecting the number of cores on t= he machine. =A0I realize this model is somewhat in conflict with Hadoop, bu= t nonetheless, that's what I'm doing. =A0Thus, for even resource ut= ilization, it would be nice to not only assign one mapper per core, but onl= y one mapper per machine. =A0I realize that if I saturate the cluster none = of this really matters, but consider the following example for clarity: 4-c= ore nodes, 10-node cluster, thus 40 slots, fully configured across mappers = and reducers (40 slots of each). =A0Say I run this program with just two ma= ppers. =A0It would run much more efficiently (in essentially half the time)= if I could force the two mappers to go to slots on two separate machines i= nstead of running the risk that Hadoop may assign them both to the same mac= hine.
>
> Can this be done?
>
> Thanks.

___________________________________________________________________________= _____
Keith Wiley =A0 =A0 kwiley@keithwi= ley.com =A0 =A0 kei= thwiley.com =A0 =A0music.keithwiley.com

"Luminous beings are we, not this crude matter."
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0-- =A0Yoda
___________________________________________________________________________= _____


--001a113497783ee09a04f1166baa-- --001a113497783ee09d04f1166bab Content-Type: image/gif; name="360.gif" Content-Transfer-Encoding: base64 Content-ID: <360@goomoji.gmail> X-Attachment-Id: 360@goomoji.gmail R0lGODlhDAAPAKIFAJh3AP/zxAAAANyuAP/gaP///wAAAAAAACH/C05FVFNDQVBFMi4wAwEAAAAh +QQJAAAFACwAAAAADAAPAAADLVi63P6wjEmrDDjrGYj/H0cIIElwpJmeQweC4hu2BGDfnnjjrLvP LtlJUikWEgAh+QQJAAAFACwAAAAADAAPAAADLVi63M4jyllGuDjHQLr3GyF8I7GNJWpaXxu2IAvM dBfS9coReLzDpspk+HAkAAAh+QQJCgAFACwAAAAADAAPAAADLVi6vPMwlhGqvS+QzXkmQhcSWTia JNWt3+qpQCxvnzynGmG/uUtOkWBjSCQmAAAh+QQJFAAFACwAAAAADAAPAAADLViz3BowyhWIvZcS gTmhnBd+Q4Vh2pmVBOC+lvbCpDmvpvopTV/8wKBwSCwWEgAh+QQJCgAFACwAAAAADAAPAAADLVi6 vPMwlhGqvS+QzXkmQhcSWTiaJNWt3+qpQCxvnzynGmG/uUtOkWBjSCQmAAAh+QQFAAAFACwAAAAA DAAPAAADLVi63M4jyllGuDjHQLr3GyF8I7GNJWpaXxu2IAvMdBfS9coReLzDpspk+HAkAAA7 --001a113497783ee09d04f1166bab--