Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id ECC9710744 for ; Thu, 12 Dec 2013 15:17:32 +0000 (UTC) Received: (qmail 52613 invoked by uid 500); 12 Dec 2013 15:17:23 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 52256 invoked by uid 500); 12 Dec 2013 15:17:23 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 52243 invoked by uid 99); 12 Dec 2013 15:17:22 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Dec 2013 15:17:22 +0000 X-ASF-Spam-Status: No, hits=3.4 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of java8964@hotmail.com designates 65.55.90.101 as permitted sender) Received: from [65.55.90.101] (HELO snt0-omc2-s26.snt0.hotmail.com) (65.55.90.101) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Dec 2013 15:17:12 +0000 Received: from SNT149-W40 ([65.55.90.71]) by snt0-omc2-s26.snt0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 12 Dec 2013 07:16:51 -0800 X-TMN: [As+W9EAOXnh1og4ODozAAV2Tr2Uke/oRuNwNq6ThuIs=] X-Originating-Email: [java8964@hotmail.com] Message-ID: Content-Type: multipart/alternative; boundary="_17bebd23-2797-4399-b85a-121514dd4014_" From: java8964 To: "user@hadoop.apache.org" Subject: RE: issue about Shuffled Maps in MR job summary Date: Thu, 12 Dec 2013 10:16:51 -0500 Importance: Normal In-Reply-To: References: ,<5DF48A23D7B14649BBA72C2F64C6663B82B356DB@szxeml523-mbx.china.huawei.com>,,,,,, MIME-Version: 1.0 X-OriginalArrivalTime: 12 Dec 2013 15:16:51.0673 (UTC) FILETIME=[2F32E490:01CEF74D] X-Virus-Checked: Checked by ClamAV on apache.org --_17bebd23-2797-4399-b85a-121514dd4014_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Or you should check your job history UI=2C which provide the similar infor= mation as job tracker=2C as you are using MR2 and Yarn. The default port of job history UI is 19888. From: java8964@hotmail.com To: user@hadoop.apache.org Subject: RE: issue about Shuffled Maps in MR job summary Date: Thu=2C 12 Dec 2013 10:06:37 -0500 =0A= =0A= =0A= Then you can check your job's status from the yarn resource manager web ui= =2C to identify what step your reducers are in. Date: Thu=2C 12 Dec 2013 11:12:47 +0800 Subject: Re: issue about Shuffled Maps in MR job summary From: justlooks@gmail.com To: user@hadoop.apache.org one of important things is my input file is very small =2Ceach file less th= an 10M=2Cand i have a huge number of files =0A= On Thu=2C Dec 12=2C 2013 at 9:58 AM=2C java8964 wrot= e: =0A= =0A= =0A= Assume the block size is 128M=2C and your mapper each finishes within half = minute=2C then there is not too much logic in your mapper=2C as it can fini= sh processing 128M around 30 seconds. If your reducers cannot finish with 1= week=2C then something is wrong. =0A= =0A= So you may need to find out following:=0A= =0A= 1) How many mappers generated in your MR job?=0A= 2) Are they all finished? (Check them in the jobtracker through web or comm= and line)=0A= 3) How many reducers in this job?=0A= 4) Are reducers starting? What stage are they in? Copying/Sorting/Reducing?= =0A= 5) If in the reducing stage=2C check the userlog of reducers. Is your code = running now? =0A= =0A= All these information you can find out from the Job Tracker web UI.=0A= =0A= Yong =0A= =0A= =0A= Date: Thu=2C 12 Dec 2013 09:03:29 +0800 =0A= =0A= Subject: Re: issue about Shuffled Maps in MR job summary From: justlooks@gmail.com To: user@hadoop.apache.org =0A= =0A= hi=2C=0A= suppose i have 5-worknode cluster=2Ceach worknode can allocate 40G mem = =2Cand i do not care map task=2Cbe cause the map task in my job finished wi= thin half a minuter=2Cas my observe the real slow task is reduce=2C i alloc= ate 12G to each reduce task=2Cso each worknode can support 3 reduce paralle= l=2Cand the whole cluster can support 15 reducer=2Cand i run the job with a= ll 15 reducer=2C and i do not know if i increase reducer number from 15 to = 30 =2Ceach reduce allocate 6G MEM=2Cthat will speed the job or not =2Cthe j= ob run on my product env=2C it run nearly 1 week=2Cit still not finished =0A= =0A= On Wed=2C Dec 11=2C 2013 at 9:50 PM=2C java8964 wrot= e: =0A= =0A= =0A= The whole job complete time depends on a lot of factors. Are you sure the r= educers part is the bottleneck? =0A= =0A= Also=2C it also depends on how many Reducer input groups it has in your MR = job. If you only have 20 reducer groups=2C even you jump your reducer count= to 40=2C then the epoch of reducers part won't have too much change=2C as = the additional 20 reducer task won't get data to process.=0A= =0A= =0A= If you have a lot of reducer input groups=2C and your cluster does have cap= acity at this time=2C and your also have a lot idle reducer slot=2C then in= crease your reducer count should decrease your whole job complete time.=0A= =0A= =0A= Make sense?=0A= =0A= Yong =0A= =0A= =0A= Date: Wed=2C 11 Dec 2013 14:20:24 +0800 Subject: Re: issue about Shuffled Maps in MR job summary From: justlooks@gmail.com To: user@hadoop.apache.org =0A= =0A= =0A= i read the doc=2C and find if i have 8 reducer =2Ca map task will output 8 = partition =2Ceach partition will be send to a different reducer=2C so if i = increase reduce number =2Cthe partition number increase =2Cbut the volume o= n network traffic is same=2Cwhy sometime =2Cincrease reducer number will no= t decrease job complete time ?=0A= =0A= =0A= On Wed=2C Dec 11=2C 2013 at 1:48 PM=2C Vinayakumar B wrote: =0A= =0A= =0A= It looks simple=2C J =0A= =20 Shuffled Maps=3D Number of Map Tasks * Number of Reducers =0A= =20 Thanks and Regards=2C =0A= Vinayakumar B =20 =0A= =0A= From: ch huang [mailto:justlooks@gmail.com]=20 =0A= Sent: 11 December 2013 10:56 To: user@hadoop.apache.org Subject: issue about Shuffled Maps in MR job summary =0A= =0A= =0A= =20 =0A= hi=2Cmaillist: =0A= i run terasort with 16 reducers and 8 reducers=2Cwhen i double r= educer number=2C the Shuffled maps is also double =2Cmy question is the job= only run 20 map tasks (total input file is 10=2Cand each file is 100M=2Cmy= block size is 64M=2Cso split is 20) why i need shuffle 160 maps in 8 reduc= ers run and 320 maps in 16 reducers run?how to caculate the shuffle maps nu= mber? =0A= =0A= =20 =0A= 16 reducer summary output: =0A= =20 =0A= =20 =0A= Shuffled Maps =3D320 =0A= =20 =0A= =0A= 8 reducer summary output: =0A= =20 =0A= Shuffled Maps =3D160 =0A= = --_17bebd23-2797-4399-b85a-121514dd4014_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Or you should check  =3Byour= job history UI=2C which provide the similar information as job tracker=2C = as you are using MR2 and Yarn.

The default port of job h= istory UI is 19888.

From: java8964@hotm= ail.com
To: user@hadoop.apache.org
Subject: RE: issue about Shuffled = Maps in MR job summary
Date: Thu=2C 12 Dec 2013 10:06:37 -0500

= =0A= =0A= =0A=
Then you can check your job's status from the yarn resourc= e manager web ui=2C to identify what step your reducers are in.

Date: Thu=2C 12 Dec 2013 11:12:47 +0800
Subj= ect: Re: issue about Shuffled Maps in MR job summary
From: justlooks@gma= il.com
To: user@hadoop.apache.org

one of important things is my i= nput file is very small =2Ceach file less than 10M=2Cand i have a huge numb= er of files

=0A=
On Thu=2C Dec 12=2C 2013 at 9:58 AM=2C java89= 64 <=3Bjava8964@hotmail.com>=3B wrote:
=0A=
=0A=
=0A=
Assume the block size is 128M=2C and your mapper each fini= shes within half minute=2C then there is not too much logic in your mapper= =2C as it can finish processing 128M around 30 seconds. If your reducers ca= nnot finish with 1 week=2C then something is wrong. =0A=

=0A=
So you may need to find out following:
=0A=

=0A=
1) How many mappers generated in your MR job?
=0A=
2) Are they all finished? (Check them in the jobtracker through web or= command line)
=0A=
3) How many reducers in this job?
=0A=
4) Are reducers starting? What stage are they in? Copying/Sorting/Redu= cing?
=0A=
5) If in the reducing stage=2C check the userlog of reducers. Is your = code running now? =3B
=0A=

=0A=
All these information you can find out from the Job Tracker web UI.=0A=

=0A=
Yong

=0A=
=0A=
=0A= Date: Thu=2C 12 Dec 2013 09:03:29 +0800 =0A=
=0A=

Subject: Re: issue about Shuffled Maps in MR job summ= ary
From: justl= ooks@gmail.com
To: user@hadoop.apache.org
=0A=
=0A=
hi=2C
=0A=
 =3B =3B =3B suppose i have 5-worknode cluster=2Ceach work= node can allocate 40G mem =2Cand i do not care map task=2Cbe cause the map = task in my job =3Bfinished within half a minuter=2Cas my observe the re= al slow task is reduce=2C i allocate 12G to each reduce task=2Cso each work= node can support 3 reduce parallel=2Cand the whole cluster can support 15 r= educer=2Cand i run the job with all 15 reducer=2C and i do not know if i in= crease reducer number from 15 to 30 =2Ceach reduce allocate 6G MEM=2Cthat w= ill speed the job or not =2Cthe job run on my product env=2C it run nearly = 1 week=2Cit still not finished
=0A=
=0A=
On Wed=2C Dec 11=2C 2013 at 9:50 PM=2C java8964 <= =3Bjava8964@hotma= il.com>=3B wrote:
=0A=
=0A=
=0A=
The whole job complete time depends on a lot of factors. A= re you sure the reducers part is the bottleneck? =0A=

=0A=
Also=2C it also depends on how many Reducer input groups it has in you= r MR job. If you only have 20 reducer groups=2C even you jump your reducer = count to 40=2C then the epoch of reducers part won't have too much change= =2C as the additional 20 reducer task won't get data to process.
=0A= =0A=

=0A=
If you have a lot of reducer input groups=2C and your cluster does hav= e capacity at this time=2C and your also have a lot idle reducer slot=2C th= en increase your reducer count should decrease your whole job complete time= .
=0A= =0A=

=0A=
Make sense?
=0A=

=0A=
Yong

=0A=
=0A=
=0A= Date: Wed=2C 11 Dec 2013 14:20:24 +0800
Subject: Re: issue about Shuffle= d Maps in MR job summary
From: justlooks@gmail.com
To: user@hadoop.apache.org =0A=
=0A=


=0A=
i read the doc=2C and find if i have 8 reducer =2Ca map task will outp= ut 8 partition =2Ceach partition will be =3Bsend to a different reducer= =2C so if i increase reduce number =2Cthe partition number increase =2Cbut = the =3Bvolume on network traffic is same=2Cwhy sometime =2Cincrease red= ucer number will not decrease job =3Bcomplete time =3B?
=0A= =0A=
 =3B
=0A=
On Wed=2C Dec 11=2C 2013 at 1:48 PM=2C Vinayakumar B <=3Bvinaya= kumar.b@huawei.com>=3B wrote:
=0A=
=0A=
=0A=
It looks simple=2C J
=0A=  =3B
Shuffled Maps=3D Number of Ma= p Tasks * Number of Reducers
= =0A=  =3B
Thanks and Reg= ards=2C
=0A= Vinayakumar B
&n= bsp=3B
=0A= =0A=
From: ch hu= ang [mailto:justlo= oks@gmail.com]
=0A= Sent: 11 December 2013 10:56
To: user@hadoop.apache.org
Subject= : issue about Shuffled Maps in MR job summary
<= /div>=0A= =0A=
=0A=
 =3B
=0A=
hi=2Cmaillist:
=0A=
 =3B =3B =3B =3B =3B =3B =3B =3B = =3B =3B i run terasort with 16 reducers and 8 reducers=2Cwhen i =3B= double reducer number=2C the Shuffled maps is also double =2Cmy question is= the job only run 20 map tasks (total input file is 10=2Cand each file is 1= 00M=2Cmy block size is 64M=2Cso split is 20) why i need shuffle 160 maps in= 8 reducers run and 320 maps in 16 reducers run?how to caculate the shuffle= maps number?
=0A=
=0A=
 =3B
=0A=
16 reducer summary output:
=0A=
 =3B
=0A=
 =3B =3B =3B
=0A=
 =3BShuffled Maps =3D320
=0A=
 =3B
=0A=
=0A=
8 =3Breducer summary output:
=0A=
 =3B
=0A=
Shuffled Maps =3D160
<= /div>

=0A=


= --_17bebd23-2797-4399-b85a-121514dd4014_--