Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3191311B23 for ; Sat, 11 May 2013 17:38:49 +0000 (UTC) Received: (qmail 17296 invoked by uid 500); 11 May 2013 17:38:44 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 16964 invoked by uid 500); 11 May 2013 17:38:44 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 16957 invoked by uid 99); 11 May 2013 17:38:44 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 May 2013 17:38:44 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dontariq@gmail.com designates 209.85.212.50 as permitted sender) Received: from [209.85.212.50] (HELO mail-vb0-f50.google.com) (209.85.212.50) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 May 2013 17:38:39 +0000 Received: by mail-vb0-f50.google.com with SMTP id w16so4224226vbb.23 for ; Sat, 11 May 2013 10:38:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:content-type; bh=4P3alinZ1Ahe77qqBJTpeyf4WE6YmoLw+e1Y6QaKArM=; b=cfhJ2q75XMN889i7/C/kFLB6Dzl7KDY3xF34aT/7O31vCf+VaS/eZ9dsQ47i6f38+h ajQxcjWTqhYaT4P1ny3yBt0LDx3VG4KaDGbjPh5uZi5bFZ25o6SABi3pXRI2Um4weZsu ZkOuwz9FUKwwoaTI6Jd7e/KGcZx4OLJY5X48yT7firn+OKW+2d8yMIW5b+DVhXaFQdvi sBKL5xo2adLeP28iQF7+KgrcUWlTIDYikPFVqt1HAsGHjiA1U3u57BbtlPLwhhLNDg9Q HLYuGqJZV6fRFJ1T27sB+iTHIhy5A496bVDMS/DilQVwI5iERHr8owJtY/TvMqoCJPE8 +YJA== X-Received: by 10.220.74.200 with SMTP id v8mr14304407vcj.60.1368293899228; Sat, 11 May 2013 10:38:19 -0700 (PDT) MIME-Version: 1.0 Received: by 10.58.152.36 with HTTP; Sat, 11 May 2013 10:37:38 -0700 (PDT) In-Reply-To: References: From: Mohammad Tariq Date: Sat, 11 May 2013 23:07:38 +0530 Message-ID: Subject: Re: Need help about task slots To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=001a11c23e04cfdc3f04dc74bf66 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c23e04cfdc3f04dc74bf66 Content-Type: text/plain; charset=ISO-8859-1 Hello guys, My 2 cents : Actually no. of mappers is primarily governed by the no. of InputSplits created by the InputFormat you are using and the no. of reducers by the no. of partitions you get after the map phase. Having said that, you should also keep the no of slots, available per slave, in mind, along with the available memory. But as a general rule you could use this approach : Take the no. of virtual CPUs*.75 and that's the no. of slots you can configure. For example, if you have 12 physical cores (or 24 virtual cores), you would have (24*.75)=18 slots. Now, based on your requirement you could choose how many mappers and reducers you want to use. With 18 MR slots, you could have 9 mappers and 9 reducers or 12 mappers and 9 reducers or whatever you think is OK with you. I don't know if it ,makes much sense, but it helps me pretty decently. Warm Regards, Tariq cloudfront.blogspot.com On Sat, May 11, 2013 at 8:57 PM, Rahul Bhattacharjee < rahul.rec.dgp@gmail.com> wrote: > Hi, > > I am also new to Hadoop world , here is my take on your question , if > there is something missing then others would surely correct that. > > For per-YARN , the slots are fixed and computed based on the crunching > capacity of the datanode hardware , once the slots per data node is > ascertained , they are divided into Map and reducer slots and that goes > into the config files and remain fixed , until changed.In YARN , its > decided at runtime based on the kind of requirement of particular task.Its > very much possible that a datanode at certain point of time running 10 > tasks and another similar datanode is only running 4 tasks. > > Coming to your question. Based of the data set size , block size of dfs > and input formater , the number of map tasks are decided , generally for > file based inputformats its one mapper per data block , however there are > way to change this using configuration settings.Reduce tasks are set using > job configuration. > > General rule as I have read from various documents is that Mappers should > run atleast a minute , so you can run a sample to find out a good size of > data block which would make you mapper run more than a minute. Now it again > depends on your SLA , in case you are not looking for a very small SLA you > can choose to run less mappers at the expense of higher runtime. > > But again its all theory , not sure how these things are handled in actual > prod clusters. > > HTH, > > > > Thanks, > Rahul > > > On Sat, May 11, 2013 at 8:02 PM, Shashidhar Rao < > raoshashidhar123@gmail.com> wrote: > >> Hi Users, >> >> I am new to Hadoop and confused about task slots in a cluster. How would >> I know how many task slots would be required for a job. Is there any >> empirical formula or on what basis should I set the number of task slots. >> >> Advanced Thanks >> > > --001a11c23e04cfdc3f04dc74bf66 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hello guys,=A0

=A0 =A0 =A0 =A0 =A0 =A0 My 2 cents :=A0

Actually no. of mappers is primarily governed by the no. of InputSplits cre= ated by the InputFormat you are using and the no. of reducers by the no. of= partitions you get after the map phase. Having said that, you should also = keep the no of slots, available per slave, in mind, along with the availabl= e memory. But as a general rule you could use this approach :

Take the no. of virtual CPUs*.75 and that's the no. of slots you can co= nfigure. For example, if you have 12 physical cores (or 24 virtual cores), = you would have (24*.75)=3D18 slots. Now, based on your requirement you coul= d choose how many mappers and reducers you want to use. With 18 MR slots, y= ou could have 9 mappers and 9 reducers or 12 mappers and 9 reducers or what= ever you think is OK with you.=A0

I don't know if it ,makes much sense, but it helps me pretty decently.<= br>


Warm Regards,
Tariq


On Sat, May 11, 2013 at 8:57 PM, Rahul B= hattacharjee <rahul.rec.dgp@gmail.com> wrote:
Hi,

I am also new to Hadoop world , here is my ta= ke on your question , if there is something missing then others would surel= y correct that.

For per-YARN , the slots are fixed and computed based on the crunch= ing capacity of the datanode hardware , once the slots per data node is asc= ertained , they are divided into Map and reducer slots and that goes into t= he config files and remain fixed , until changed.In YARN , its decided at r= untime based on the kind of requirement of particular task.Its very much po= ssible that a datanode at certain point of time running=A0 10 tasks and ano= ther similar datanode is only running 4 tasks.

Coming to your question. Based of the data set size , block size of= dfs and input formater , the number of map tasks are decided , generally f= or file based inputformats its one mapper per data block , however there ar= e way to change this using configuration settings.Reduce tasks are set usin= g job configuration.

General rule as I have read from various documents is that Mappers = should run atleast a minute , so you can run a sample to find out a good si= ze of data block which would make you mapper run more than a minute. Now it= again depends on your SLA , in case you are not looking for a very small S= LA you can choose to run less mappers at the expense of higher runtime.

But again its all theory , not sure how these things are handled in= actual prod clusters.

HTH,



Thanks,
Rahul


On Sat, May 11, = 2013 at 8:02 PM, Shashidhar Rao <raoshashidhar123@gmail.com&g= t; wrote:
Hi Users,

=
I am new to Hadoop and confused about task slots in a cluster. How wo= uld I know how many task slots would be required for a job. Is there any em= pirical formula or on what basis should I set the number of task slots.

Advanced Thanks


--001a11c23e04cfdc3f04dc74bf66--