Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F34301008E for ; Tue, 25 Feb 2014 20:40:25 +0000 (UTC) Received: (qmail 90498 invoked by uid 500); 25 Feb 2014 20:40:16 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 90332 invoked by uid 500); 25 Feb 2014 20:40:16 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 90312 invoked by uid 99); 25 Feb 2014 20:40:15 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Feb 2014 20:40:15 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dontariq@gmail.com designates 209.85.220.174 as permitted sender) Received: from [209.85.220.174] (HELO mail-vc0-f174.google.com) (209.85.220.174) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Feb 2014 20:40:11 +0000 Received: by mail-vc0-f174.google.com with SMTP id im17so7760332vcb.5 for ; Tue, 25 Feb 2014 12:39:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=6+rBCOvLBKcUI9SQmmQVzzlEDsrnitnpRp00XsB2Tgw=; b=uTIXIKK6s2i2Ecy2M3UN2tEePhyTqJ5DtZHz1Pbly/wHniLLJzOI5keq7eT8xjeMq8 IYqiAS0At8YaFO+Rpq8BVLP2uRXKc+8oLDID2estEUh9uwP74PEaV3u4liBRiXnx+Ci7 80tr4H8NV9Sc7M/itdgtTGhZguVtjQGly8VeRZGczEMbDfsvT1XyNvFJzrib8uabqQap u1RU0gBPPNFlNKP7Xe1oZ8zV7YJ3mdfh+oZ2FFojwO0BNtg8kkfwmx/pb2tXlv13aNeb kj8Q4lkz4ooepMj7M0nZQ9qtVSJHWfi8ipqknMZI/XsH+L/6iyFNyDcpNPaRRiHRcMvs uhxQ== X-Received: by 10.58.170.69 with SMTP id ak5mr2613338vec.28.1393360790731; Tue, 25 Feb 2014 12:39:50 -0800 (PST) MIME-Version: 1.0 Received: by 10.58.88.70 with HTTP; Tue, 25 Feb 2014 12:39:10 -0800 (PST) In-Reply-To: References: From: Mohammad Tariq Date: Wed, 26 Feb 2014 02:09:10 +0530 Message-ID: Subject: Re: Mappers vs. Map tasks To: "hdfs-user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=047d7b86e2d6f9c0ec04f341165e X-Virus-Checked: Checked by ClamAV on apache.org --047d7b86e2d6f9c0ec04f341165e Content-Type: text/plain; charset=ISO-8859-1 Hi Sugandha, Please find my comments embedded below : No. of mappers are decided as: Total_File_Size/Max. Block Size. Thus, if the file is smaller than the block size, only one mapper will be invoked. Right? This is true(but not always). The basic criteria behind map creation is the logic inside *getSplits* method of *InputFormat* being used in your MR job. It is the behavior of *file based InputFormats*, typically sub-classes of *FileInputFormat*, to split the input data into splits based on the total size, in bytes, of the input files. See *this*for more details. And yes, if the file is smaller than the block size then only 1 mapper will be created. If yes, it means, the map() will be called only once. Right? In this case, if there are two datanodes with a replication factor as 1: only one datanode(mapper machine) will perform the task. Right? A mapper is called for each split. Don't get confused with the MR's split and HDFS's block. Both are different(They may overlap though, as in case of FileInputFormat). HDFS blocks are physical partitioning of your data, while an InputSplit is just a logical partitioning. If you have a file which is smaller than the HDFS blocksize then only one split will be created, hence only 1 mapper will be called. And this will happen on the node where this file resides. The map() function is called by all the datanodes/slaves right? If the no. of mappers are more than the no. of slaves, what happens? map() doesn't get called by anybody. It rather gets created on the node where the chunk of data to be processed resides. A slave node can run multiple mappers based on the availability of CPU slots. One more thing to ask: No. of blocks = no. of mappers. Thus, those many no. of times the map() function will be called right? No. of blocks = no. of splits = no. of mappers. A map is called only once per split per node where that split is present. HTH Warm Regards, Tariq cloudfront.blogspot.com On Tue, Feb 25, 2014 at 3:54 PM, Sugandha Naolekar wrote: > Hi Bertrand, > > As you said, no. of HDFS blocks = no. of input splits. But this is only > true when you set isSplittable() as false or when your input file size is > less than the block size. Also, when it comes to text files, the default > textinputformat considers each line as one input split which can be then > read by RecordReader in K,V format. > > Please correct me if I don't make sense. > > -- > Thanks & Regards, > Sugandha Naolekar > > > > > > On Tue, Feb 25, 2014 at 2:07 PM, Bertrand Dechoux wrote: > >> The wiki (or Hadoop The Definitive Guide) are good ressources. >> >> https://www.inkling.com/read/hadoop-definitive-guide-tom-white-3rd/chapter-7/input-formats >> >> Mapper is the name of the abstract class/interface. It does not really >> make sense to talk about number of mappers. >> A task is a jvm that can be launched only if there is a free slot ie for >> a given slot, at a given time, there will be at maximum only a single task. >> During the task, the configured Mapper will be instantiated. >> >> Always : >> Number of input splits = no. of map tasks >> >> And generally : >> number of hdfs blocks = number of input splits >> >> Regards >> >> Bertrand >> >> PS : I don't know if it is only my client, but avoid red when writting a >> mail. >> >> On Tue, Feb 25, 2014 at 8:49 AM, Dieter De Witte wrote: >> >>> Each node has a tasktracker with a number of map slots. A map slot hosts >>> as mapper. A mapper executes map tasks. If there are more map tasks than >>> slots obviously there will be multiple rounds of mapping. >>> >>> The map function is called once for each input record. A block is >>> typically 64MB and can contain a multitude of record, therefore a map task >>> = run the map() function on all records in the block. >>> >>> Number of blocks = no. of map tasks (not mappers) >>> >>> Furthermore you have to make a distinction between the two layers. You >>> have a layer for computations which consists of a jobtracker and a set of >>> tasktrackers. The other layer is responsible for storage. The HDFS has a >>> namenode and a set of datanodes. >>> >>> In mapreduce the code is executed where the data is. So if a block is in >>> datanode 1, 2 and 3, then the map task associated with this block will >>> likely be executed on one of those physical nodes, by tasktracker 1, 2 or >>> 3. But this is not necessary, thing can be rearranged. >>> >>> Hopefully this gives you a little more insigth. >>> >>> Regards, Dieter >>> >>> >>> 2014-02-25 7:05 GMT+01:00 Sugandha Naolekar : >>> >>> One more thing to ask: No. of blocks = no. of mappers. Thus, those many >>>> no. of times the map() function will be called right? >>>> >>>> -- >>>> Thanks & Regards, >>>> Sugandha Naolekar >>>> >>>> >>>> >>>> >>>> >>>> On Tue, Feb 25, 2014 at 11:27 AM, Sugandha Naolekar < >>>> sugandha.n87@gmail.com> wrote: >>>> >>>>> Hello, >>>>> >>>>> As per the various articles I went through till date, the File(s) are >>>>> split in chunks/blocks. On the same note, would like to ask few things: >>>>> >>>>> >>>>> 1. No. of mappers are decided as: Total_File_Size/Max. Block Size. >>>>> Thus, if the file is smaller than the block size, only one mapper will be >>>>> invoked. Right? >>>>> 2. If yes, it means, the map() will be called only once. Right? In >>>>> this case, if there are two datanodes with a replication factor as 1: only >>>>> one datanode(mapper machine) will perform the task. Right? >>>>> 3. The map() function is called by all the datanodes/slaves right? >>>>> If the no. of mappers are more than the no. of slaves, what happens? >>>>> >>>>> -- >>>>> Thanks & Regards, >>>>> Sugandha Naolekar >>>>> >>>>> >>>>> >>>>> >>>> >>> >> > --047d7b86e2d6f9c0ec04f341165e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi Sugandha,

Please find my comments em= bedded below :

=A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 No. of mappers are decided as: Total_File_Size/Max. Block Size.= Thus, if the file is smaller than the block size, only one mapper will be = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 invoked. Right?=
=A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0=A0This is true(but not always). The basic criteria behind map creation is t= he logic inside getSplits method of InputFormat being used in= your =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 MR job. It is the behavior of= file based InputFormats, typically sub-classes of FileInputForma= t, to split the input data into splits based =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 on the total size, in bytes, of the input files. See this for more details. And = yes, if the file is smaller than the block size then only 1 mapper will =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 be created.

=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= If yes, it means, the map() will be called only once. Right? In this case,= if there are two datanodes with a replication factor as 1: only one =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 datanode(mapper mac= hine) will perform the task. Right?
=A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0=A0A = mapper is called for each split. Don't get confused with the MR's s= plit and HDFS's block. Both are different(They may overlap though, as i= n =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 case of FileInputFormat). HDFS bl= ocks are physical partitioning of your data, while an InputSplit is just a = logical partitioning. If you have a =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 file which is smaller than the HDFS blocksize then only one split will= be created, hence only 1 mapper will be called. And this will happen on = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 the node where this file resides.

= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 The map() function is called by all th= e datanodes/slaves right? If the no. of mappers are more than the no. of sl= aves, what happens?
=A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 map() doesn't get called by anybody. It rather gets= created on the node where the chunk of data to be processed resides. A sla= ve node can run =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 multiple mapper= s based on the availability of CPU slots.

=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0One more th= ing to ask: No. of blocks =3D no. of mappers. Thus, those many no. of times= the map() function will be called right?
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0No. of blocks =3D no. o= f splits =3D no. of mappers. A map is called only once per split per node w= here that split is present.

HTH

Warm Regards,
Tariq
<= a href=3D"http://cloudfront.blogspot.com" target=3D"_blank">cloudfront.blog= spot.com


On Tue, Feb 25, 2014 at 3:54 PM, Sugandh= a Naolekar <sugandha.n87@gmail.com> wrote:
Hi Bertrand,

As you said, no. of HDFS blocks =3D=A0 no. of input splits. But this is on= ly true when you set isSplittable() as false or when your input file size i= s less than the block size. Also, when it comes to text files, the default = textinputformat considers each line as one input split which can be then re= ad by RecordReader in K,V format.

Please correct me if I don't make sense.

--
Thanks &=A0Regards,
Sugandha Naolekar





On Tue, Feb 25, 2014 at = 2:07 PM, Bertrand Dechoux <dechouxb@gmail.com> wrote:
The wiki (or Hadoop The Definitive Guide) are good ressour= ces.
https://www.inkling.= com/read/hadoop-definitive-guide-tom-white-3rd/chapter-7/input-formats<= br>

Mapper is the name of the abstract class/interface. It = does not really make sense to talk about number of mappers.
A tas= k is a jvm that can be launched only if there is a free slot ie for a given= slot, at a given time, there will be at maximum only a single task. During= the task, the configured Mapper will be instantiated.

Always :
Number of input splits =3D no. of map tasks<= br>

And generally :
number of hdfs block= s =3D number of input splits

Regards

Bertrand

PS : I don= 9;t know if it is only my client, but avoid red when writting a mail.

On Tue, Feb 25, 2014 at 8:49 AM, Dieter De Witte <drdwitte@gmail.com&= gt; wrote:
Each node has a tasktracker with = a number of map slots. A map slot hosts as mapper. A mapper executes map ta= sks. If there are more map tasks than slots obviously there will be multipl= e rounds of mapping.

The map function is called once for each input record. A block is= typically 64MB and can contain a multitude of record, therefore a map task= =3D run the map() function on all records in the block.

Numbe= r of blocks =3D no. of map tasks (not mappers)

Furthermore you have to make a distinction between the two layers= . You have a layer for computations which consists of a jobtracker and a se= t of tasktrackers. The other layer is responsible for storage. The HDFS has= a namenode and a set of datanodes.

In mapreduce the code is executed where the data is. So if a bloc= k is in datanode 1, 2 and 3, then the map task associated with this block w= ill likely be executed on one of those physical nodes, by tasktracker 1, 2 = or 3. But this is not necessary, thing can be rearranged.

Hopefully this gives you a little more insigth.

Regards, D= ieter


2014-02-25 7:05 GMT+01:00 Sugandha Naolekar <sugandha.n87@gmail.com= >:

One more thing to ask: No. of blocks =3D no. of mappers. Thus, those many n= o. of times the map() function will be called right?

--
Thanks &=A0Regards,
Sugandha Naolekar





On Tue, Feb 25, 2014 at = 11:27 AM, Sugandha Naolekar <sugandha.n87@gmail.com> wr= ote:
Hello,

As per the various articles I went through till date, the File(s) ar= e split in chunks/blocks. On the same note, would like to ask few things:
  1. No. of mappers are decided as: Total_File_Size/Max. Block Size.= Thus, if the file is smaller than the block size, only one mapper will be = invoked. Right?
  2. If yes, it means, the map() will be called only onc= e. Right? In this case, if there are two datanodes with a replication facto= r as 1: only one datanode(mapper machine) will perform the task. Right?
  3. The map() function is called by all the datanodes/slaves right? If the = no. of mappers are more than the no. of slaves, what happens?
=
--
Thanks &=A0Regards,
Sugandha Naole= kar








--047d7b86e2d6f9c0ec04f341165e--