Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5CA5D10ADA for ; Wed, 11 Dec 2013 22:30:36 +0000 (UTC) Received: (qmail 68066 invoked by uid 500); 11 Dec 2013 22:30:31 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 67968 invoked by uid 500); 11 Dec 2013 22:30:31 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 67959 invoked by uid 99); 11 Dec 2013 22:30:31 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Dec 2013 22:30:31 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of kawa.adam@gmail.com designates 209.85.223.182 as permitted sender) Received: from [209.85.223.182] (HELO mail-ie0-f182.google.com) (209.85.223.182) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Dec 2013 22:30:24 +0000 Received: by mail-ie0-f182.google.com with SMTP id as1so12216392iec.27 for ; Wed, 11 Dec 2013 14:30:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=Rjnz+47Cbgsm6qclKAGVV7zc+opV8UL/iW3ArUGrF1I=; b=ywvOtcs0Vz8YrEIk6jFunuoS7CIIbGP5tXnIEYOn7MFcFjhnzMuMIDxkL3L1JjFtCK dGUN5XOsb/OVbDt8kmFOnJfWfxfQCxyYA5zdzmbrGYmgfXkv6+lL+/QvF3tLiQDjMceS v+PqSg+mNUosvfJnc3zo3iy7tm+18093vrTByjENVNTwCvve7lJFcwU8VLxcsrLqahvo n4KJETamEsnxLdyaNqHLjDdX+rV9U3CN47wD5YYvQ4jfHrj4HmWdX1z2WtEY7yNQ69+W 5QVJgW2bQFWgmPLOpF3XeonIQnIjoMp6ALfMEV1nWfgBjWlnwqYDlOvkILOGQOe87oe+ B8tA== MIME-Version: 1.0 X-Received: by 10.43.154.73 with SMTP id ld9mr3429485icc.53.1386801003614; Wed, 11 Dec 2013 14:30:03 -0800 (PST) Received: by 10.42.153.136 with HTTP; Wed, 11 Dec 2013 14:30:03 -0800 (PST) In-Reply-To: References: <5DF48A23D7B14649BBA72C2F64C6663B82B356DB@szxeml523-mbx.china.huawei.com> Date: Wed, 11 Dec 2013 23:30:03 +0100 Message-ID: Subject: Re: issue about Shuffled Maps in MR job summary From: Adam Kawa To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a11c3078031d76d04ed49c5fd X-Virus-Checked: Checked by ClamAV on apache.org --001a11c3078031d76d04ed49c5fd Content-Type: text/plain; charset=ISO-8859-1 > why sometime ,increase reducer number will not decrease job complete time ? Apart from valid information that Yong wrote in the previous point, please note that: 1) You do not want to have very shortly lived (seconds) reduce tasks, because the overhead for coordinating them, starting JVMs, setting up the connections to all map tasks becomes too costly. It depends on your use case, but usually MapReduce jobs are for batch processing, and at my company we set the number of reduce tasks to make sure that each task runs at least a couple of minutes (for production jobs that are scheduled in "background", we aim for ~10 minutes). 2) We you have more reduce tasks, then you need more slots (or containers, if you use YARN). Sometimes, you can not get slots/containers as quick as you want, so that you can get stuck waiting for more resources. Then job completion time extends. 3) It you have thinner reducers, then they probably they write smaller output files to HDFS. Small files are problematic for HDFS (e.g. higher memory requirement on NN, bigger load on NN, slower NN restarts, more random than streaming access pattern and more). If the output of that job is later processed by another job, then you will see thin mappers (this can be partially alleviated by CombineFileInputFormat, though). 2013/12/11 java8964 > The whole job complete time depends on a lot of factors. Are you sure the > reducers part is the bottleneck? > > Also, it also depends on how many Reducer input groups it has in your MR > job. If you only have 20 reducer groups, even you jump your reducer count > to 40, then the epoch of reducers part won't have too much change, as the > additional 20 reducer task won't get data to process. > > If you have a lot of reducer input groups, and your cluster does have > capacity at this time, and your also have a lot idle reducer slot, then > increase your reducer count should decrease your whole job complete time. > > Make sense? > > Yong > > ------------------------------ > Date: Wed, 11 Dec 2013 14:20:24 +0800 > Subject: Re: issue about Shuffled Maps in MR job summary > From: justlooks@gmail.com > To: user@hadoop.apache.org > > > i read the doc, and find if i have 8 reducer ,a map task will output 8 > partition ,each partition will be send to a different reducer, so if i > increase reduce number ,the partition number increase ,but the volume on > network traffic is same,why sometime ,increase reducer number will not > decrease job complete time ? > > On Wed, Dec 11, 2013 at 1:48 PM, Vinayakumar B wrote: > > It looks simple, J > > > > Shuffled Maps= Number of Map Tasks * Number of Reducers > > > > Thanks and Regards, > > Vinayakumar B > > > > *From:* ch huang [mailto:justlooks@gmail.com] > *Sent:* 11 December 2013 10:56 > *To:* user@hadoop.apache.org > *Subject:* issue about Shuffled Maps in MR job summary > > > > hi,maillist: > > i run terasort with 16 reducers and 8 reducers,when i double > reducer number, the Shuffled maps is also double ,my question is the job > only run 20 map tasks (total input file is 10,and each file is 100M,my > block size is 64M,so split is 20) why i need shuffle 160 maps in 8 reducers > run and 320 maps in 16 reducers run?how to caculate the shuffle maps number? > > > > 16 reducer summary output: > > > > > > Shuffled Maps =320 > > > > 8 reducer summary output: > > > > Shuffled Maps =160 > > > --001a11c3078031d76d04ed49c5fd Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
>=A0why sometime ,increase reducer number will not decrease job=A0comp= lete time=A0?

Apart from valid information that=A0Yong wrote in the previous point, please not= e that:

1) You do not want to have very shortly lived (seconds) r= educe tasks, because the overhead for=A0coordinating them, starting JVMs, setting up the connectio= ns to all map tasks becomes too costly. It depends on your use case, but us= ually MapReduce jobs are for batch processing, and at my company we set the= number of reduce tasks to make sure that each task runs at least a couple = of minutes (for production jobs that are scheduled in "background"= ;, we aim for ~10 minutes).

2) We you have more reduce tasks, then you need more slot= s (or containers, if you use YARN). Sometimes, you can not get slots/contai= ners as quick as you want, so that you can get stuck=A0waiting=A0for more r= esources. Then job completion time extends.

3) It you have=A0thinner=A0reducers, then they probably t= hey write smaller output files to HDFS. Small files are problematic for HDF= S (e.g. higher memory requirement on NN, bigger load on NN,=A0slower NN res= tarts, more random than streaming access pattern and more). If the output of that job is later pro= cessed by another job, then you will see thin mappers (this can be partiall= y alleviated by CombineFileInputFormat, though).


2013/12= /11 java8964 <java8964@hotmail.com>
The whole job complete time depends on a lot of facto= rs. Are you sure the reducers part is the bottleneck?

Al= so, it also depends on how many Reducer input groups it has in your MR job.= If you only have 20 reducer groups, even you jump your reducer count to 40= , then the epoch of reducers part won't have too much change, as the ad= ditional 20 reducer task won't get data to process.

If you have a lot of reducer input groups, and your clu= ster does have capacity at this time, and your also have a lot idle reducer= slot, then increase your reducer count should decrease your whole job comp= lete time.

Make sense?

Yong

<= hr>Date: Wed, 11 Dec 2013 14:20:24 +0800
Subject: Re: issue about Shuffl= ed Maps in MR job summary
From: justlooks@gmail.com
To: user@hadoop= .apache.org


i read the doc, and find= if i have 8 reducer ,a map task will output 8 partition ,each partition wi= ll be=A0send to a different reducer, so if i increase reduce number ,the pa= rtition number increase ,but the=A0volume on network traffic is same,why so= metime ,increase reducer number will not decrease job=A0complete time=A0?
=A0
On Wed, Dec 11, 2013 at 1:48 PM, Vinayakumar B <<= a href=3D"mailto:vinayakumar.b@huawei.com" target=3D"_blank">vinayakumar.b@= huawei.com> wrote:

It looks simple, J

=A0

Shuffled Maps=3D Number of Map Tasks * Number of Reducers=

=A0

Thanks and Regards,

Vinayakumar B

=A0

From: ch huang [mailto:justlooks@gmail.com]
Sent: 11 December 2013 10:56
To: user@hadoop.apache.org
Subject= : issue about Shuffled Maps in MR job summary

=A0

hi,maillist:

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 i run terasort with 16 reducers and 8 red= ucers,when i=A0double reducer number, the Shuffled maps is also double ,my = question is the job only run 20 map tasks (total input file is 10,and each = file is 100M,my block size is 64M,so split is 20) why i need shuffle 160 ma= ps in 8 reducers run and 320 maps in 16 reducers run?how to caculate the sh= uffle maps number?

=A0

16 reducer summary output:

=A0

=A0=A0=A0

=A0Shuffled Maps =3D320

=A0

8=A0reducer summary output:

=A0

Shuffled Maps =3D160

<= /div>


--001a11c3078031d76d04ed49c5fd--