Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id ED933E271 for ; Tue, 29 Jan 2013 20:08:54 +0000 (UTC) Received: (qmail 94805 invoked by uid 500); 29 Jan 2013 20:08:50 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 94514 invoked by uid 500); 29 Jan 2013 20:08:50 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 94507 invoked by uid 99); 29 Jan 2013 20:08:50 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jan 2013 20:08:50 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of vinodkv@hortonworks.com designates 209.85.214.178 as permitted sender) Received: from [209.85.214.178] (HELO mail-ob0-f178.google.com) (209.85.214.178) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jan 2013 20:08:43 +0000 Received: by mail-ob0-f178.google.com with SMTP id wd20so853486obb.37 for ; Tue, 29 Jan 2013 12:08:22 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type:x-gm-message-state; bh=EV2YEgBv5C+Mb5h8q0ce/MGcJMkgTHKc1Qh4igycY/M=; b=J2UJh8el/zvrzzQqEyRERsbB/ZFfVAO3cNPxoT099pkGXItDUc2NdQ3NMIQW8lzhA6 kAe+xCx+xW9XpaIKAiKuIw1LNTQ3ZHeD0DBkF95YkB2x0KV/rE9wTagifL2hExfqWgZJ OrK4cL2i+VrjTMy1H6bf6Zbf1LSFVXA9xVWTJPkHscHGpVd5JvNYfrCvn3djrjU0EAkx klucmbourCvBN9zdPXR2Z3qdZEKKN5IVphnw00haF/HS8eTdV72g/9PnXL3bMH2TZI8P 63+tF6FDKGK989bUFOv/2KEeuk393de7oSAvQdgxycv8lOIy1WyKGSlB3YpCKJ4MyY99 GpQA== MIME-Version: 1.0 X-Received: by 10.182.111.34 with SMTP id if2mr1776534obb.6.1359490102749; Tue, 29 Jan 2013 12:08:22 -0800 (PST) Received: by 10.60.15.130 with HTTP; Tue, 29 Jan 2013 12:08:22 -0800 (PST) In-Reply-To: References: <7F7E1558-2A7C-487A-AE0F-6CDCEE1EE3E8@apache.org> Date: Tue, 29 Jan 2013 12:08:22 -0800 Message-ID: Subject: Re: number of mapper tasks From: Vinod Kumar Vavilapalli To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=14dae9399495a66c1104d472f4d4 X-Gm-Message-State: ALoCoQmEDQdqNktXuSldSUKxKLJwrDGYxKfnuuTl8C+wTUul5PlYAxlCIl1OUgMdBAvJNQlmT/oz X-Virus-Checked: Checked by ClamAV on apache.org --14dae9399495a66c1104d472f4d4 Content-Type: text/plain; charset=ISO-8859-1 Tried looking at your code, it's a bit involved. Instead of trying to run the job, try unit-testing your input format. Test for getSplits(), whatever number of splits that method returns, that will be the number of mappers that will run. You can also use LocalJobRunner also for this - set mapred.job.tracker to local and run your job locally on your machine instead of trying on a cluster. HTH, +Vinod On Tue, Jan 29, 2013 at 4:53 AM, Marcelo Elias Del Valle wrote: > Hello, > > I have been able to make this work. I don't know why, but when but > input file is zipped (read as a input stream) it creates only 1 mapper. > However, when it's not zipped, it creates more mappers (running 3 instances > it created 4 mappers and running 5 instances, it created 8 mappers). > I really would like to know why this happens and even with this number > of mappers, I would like to know why more mappers aren't created. I was > reading part of the book "Hadoop - The definitive guide" ( > https://www.inkling.com/read/hadoop-definitive-guide-tom-white-3rd/chapter-7/input-formats) > which says: > > "The JobClient calls the getSplits() method, passing the desired number > of map tasks as the numSplits argument. This number is treated as a hint, > as InputFormat implementations are free to return a different number of > splits to the number specified in numSplits. Having calculated the > splits, the client sends them to the jobtracker, which uses their storage > locations to schedule map tasks to process them on the tasktrackers. ..." > > I am not sure on how to get more info. > > Would you recommend me to try to find the answer on the book? Or > should I read hadoop source code directly? > > Best regards, > Marcelo. > > > 2013/1/29 Marcelo Elias Del Valle > >> I implemented my custom input format. Here is how I used it: >> >> https://github.com/mvallebr/CSVInputFormat/blob/master/src/test/java/org/apache/hadoop/mapreduce/lib/input/test/CSVTestRunner.java >> >> As you can see, I do: >> importerJob.setInputFormatClass(CSVNLineInputFormat.class); >> >> And here is the Input format and the linereader: >> >> https://github.com/mvallebr/CSVInputFormat/blob/master/src/main/java/org/apache/hadoop/mapreduce/lib/input/CSVNLineInputFormat.java >> >> https://github.com/mvallebr/CSVInputFormat/blob/master/src/main/java/org/apache/hadoop/mapreduce/lib/input/CSVLineRecordReader.java >> >> In this input format, I completely ignore these other parameters and get >> the splits by the number of lines. The amount of lines per map can be >> controlled by the same parameter used in NLineInputFormat: >> >> public static final String LINES_PER_MAP = >> "mapreduce.input.lineinputformat.linespermap"; >> However, it has really no effect on the number of maps. >> >> >> >> 2013/1/29 Vinod Kumar Vavilapalli >> >>> >>> Regarding your original question, you can use the min and max split >>> settings to control the number of maps: >>> http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.html. See #setMinInputSplitSize and #setMaxInputSplitSize. Or >>> use mapred.min.split.size directly. >>> >>> W.r.t your custom inputformat, are you sure you job is using this >>> InputFormat and not the default one? >>> >>> HTH, >>> +Vinod Kumar Vavilapalli >>> Hortonworks Inc. >>> http://hortonworks.com/ >>> >>> On Jan 28, 2013, at 12:56 PM, Marcelo Elias Del Valle wrote: >>> >>> Just to complement the last question, I have implemented the getSplits >>> method in my input format: >>> >>> https://github.com/mvallebr/CSVInputFormat/blob/master/src/main/java/org/apache/hadoop/mapreduce/lib/input/CSVNLineInputFormat.java >>> >>> However, it still doesn't create more than 2 map tasks. Is there >>> something I could do about it to assure more map tasks are created? >>> >>> Thanks >>> Marcelo. >>> >>> >>> 2013/1/28 Marcelo Elias Del Valle >>> >>>> Sorry for asking too many questions, but the answers are really >>>> happening. >>>> >>>> >>>> 2013/1/28 Harsh J >>>> >>>>> This seems CPU-oriented. You probably want the NLineInputFormat? See >>>>> >>>>> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/lib/NLineInputFormat.html >>>>> . >>>>> This should let you spawn more maps as we, based on your N factor. >>>>> >>>> >>>> Indeed, CPU is my bottleneck. That's why I want more things in parallel. >>>> Actually, I wrote my own InputFormat, to be able to process multiline >>>> CSVs: https://github.com/mvallebr/CSVInputFormat >>>> I could change it to read several lines at a time, but would this alone >>>> allow more tasks running in parallel? >>>> >>>> >>>>> Not really - "Slots" are capacities, rather than split factors >>>>> themselves. You can have N slots always available, but your job has to >>>>> supply as many map tasks (based on its input/needs/etc.) to use them >>>>> up. >>>>> >>>> >>>> But how can I do that (supply map tasks) in my job? changing its code? >>>> hadoop config? >>>> >>>> >>>>> Unless your job sets the number of reducers to 0 manually, 1 default >>>>> reducer is always run that waits to see if it has any outputs from >>>>> maps. If it does not receive any outputs after maps have all >>>>> completed, it dies out with behavior equivalent to a NOP. >>>>> >>>> Ok, I did job.setNumReduceTasks(0); , guess this will solve this part, >>>> thanks! >>>> >>>> >>>> -- >>>> Marcelo Elias Del Valle >>>> http://mvalle.com - @mvallebr >>>> >>> >>> >>> >>> -- >>> Marcelo Elias Del Valle >>> http://mvalle.com - @mvallebr >>> >>> >>> >> >> >> -- >> Marcelo Elias Del Valle >> http://mvalle.com - @mvallebr >> > > > > -- > Marcelo Elias Del Valle > http://mvalle.com - @mvallebr > -- +Vinod Hortonworks Inc. http://hortonworks.com/ --14dae9399495a66c1104d472f4d4 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

Tried looking at your code, it= 9;s a bit involved. Instead of trying to run the job, try unit-testing your= input format. Test for getSplits(), whatever number of splits that method = returns, that will be the number of mappers that will run.

You can also use LocalJobRunner also for this - set mapred.job.tracker = to local and run your job locally on your machine instead of trying on a cl= uster.

HTH,
+Vinod



On Tue, Jan 29, 2013 at 4:53 AM, Marcelo= Elias Del Valle <mvallebr@gmail.com> wrote:
Hello,=A0

=A0 =A0 I have been able to m= ake this work. I don't know why, but when but input file is zipped (rea= d as a input stream) it creates only 1 mapper. However, when it's not z= ipped, it creates more mappers (running 3 instances it created 4 mappers an= d running 5 instances, it created 8 mappers).
=A0 =A0 I really would like to know why this happens and even with thi= s number of mappers, I would like to know why more mappers aren't creat= ed. I was reading part of the book "Hadoop - The definitive guide"= ; (https://www.inkling.com/re= ad/hadoop-definitive-guide-tom-white-3rd/chapter-7/input-formats) which= says:=A0

"The=A0JobC= lient=A0calls the=A0getSplits= ()=A0method, passing the desired number of map tasks as th= e=A0numSplits=A0argument. Thi= s number is treated as a hint, as=A0InputForma= t=A0implementations are free to return a different number = of splits to the number specified in=A0numSpli= ts. Having calculated the splits, the client sends them to= the jobtracker, which uses their storage locations to schedule map tasks t= o process them on the tasktrackers. ..."
=A0 =A0=A0
=A0 =A0 =A0I am not sure on how to get more info.= =A0

=A0 =A0 =A0Would you recommend me to try to fi= nd the answer on the book? Or should I read hadoop source code directly?

Best regards,
Marcelo.


2013/1/29 Marcelo Elias Del Valle <mvallebr@gmail.com>
I implemented my custom inp= ut format. Here is how I used it:

As you can see, I do:=A0
importerJob.setInputFormatClass(CSVNLineInpu= tFormat.class);

And here is the Input format and the linereader:<= /div>

In this input format, I completely ignore these o= ther parameters and get the splits by the number of lines. The amount of li= nes per map can be controlled by the same parameter used in NLineInputForma= t:

= public static final Str= ing LINES_PER_MAP =3D= "mapreduce.i= nput.lineinputformat.linespermap";
However, it has really no effect on the number of maps.



2013/1/29 Vinod Kumar Vavilapalli <vinodkv@hortonwo= rks.com>
Regarding your original question, you can use the min and max s= plit settings to control the number of maps: http://hadoop.apache.org/docs/stable/api/org/apach= e/hadoop/mapreduce/lib/input/FileInputFormat.html . See #setMinInputSpl= itSize and #setMaxInputSplitSize. Or use=A0mapred.min.split.size directly.<= /div>

W.r.t your custom inputformat, are you sure you job is using= this InputFormat and not the default one?

HTH,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
= http://hortonworks.co= m/

On Jan 28, 2013, at 12:56 PM, Marcelo Elias Del Valle wrote:<= /div>
Just to complement the = last question, I have implemented the=A0getSplits method in my input format= :

However, it still doesn't create more than 2 = map tasks. Is there something I could do about it to assure more map tasks = are created?

Thanks
Marcelo.


2013/1/28 Marcelo Elias Del Valle <mvallebr@gmail.com>
Sorry for asking too many q= uestions, but the answers are really happening.
<= br>
2013/1/28 Harsh J <harsh@cloudera.com= >
This seems CPU-oriented. You probably want the NLineInputF= ormat? See
http://hadoop.apache= .org/common/docs/current/api/org/apache/hadoop/mapred/lib/NLineInputFormat.= html.
This should let you spawn more maps as we, based on your N factor.

Indeed, CPU is my bottleneck. That's = why I want more things in parallel.
Actually, I wrote my own Inpu= tFormat, to be able to process multiline CSVs:=A0https://github.com/mvallebr/= CSVInputFormat
I could change it to read several lines at a time, but would this alon= e allow more tasks running in parallel?
=A0
Not really - "Slots" are capacities, rather than split facto= rs
themselves. You can have N slots always available, but your job has to
supply as many map tasks (based on its input/needs/etc.) to use them
up.

But how can I do that (supply= map tasks) in my job? changing its code? hadoop config?
=A0=
Unless your job sets the number of reducers to 0 manually, 1 default
reducer is always run that waits to see if it has any outputs from
maps. If it does not receive any outputs after maps have all
completed, it dies out with behavior equivalent to a NOP.
<= /div>
Ok, I did job.setNumReduceTasks(0); , guess this will solve this= part, thanks!


--
Marcelo Elias De= l Valle
http://mvalle.com=A0- = @mvallebr



--
Marcelo Elia= s Del Valle
http://mval= le.com=A0- @mvallebr




--
Marcelo Elias Del Valle
http://mvalle.com=A0- @mvallebr



--
= Marcelo Elias Del Valle
= http://mvalle.com=A0- @mvallebr


--
+Vinod
H= ortonworks Inc.
ht= tp://hortonworks.com/ --14dae9399495a66c1104d472f4d4--