Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 142A0F557 for ; Fri, 12 Apr 2013 04:57:57 +0000 (UTC) Received: (qmail 3608 invoked by uid 500); 12 Apr 2013 04:57:52 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 3207 invoked by uid 500); 12 Apr 2013 04:57:48 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 3199 invoked by uid 99); 12 Apr 2013 04:57:48 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Apr 2013 04:57:48 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of kshanthi501@gmail.com designates 209.85.223.179 as permitted sender) Received: from [209.85.223.179] (HELO mail-ie0-f179.google.com) (209.85.223.179) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Apr 2013 04:57:43 +0000 Received: by mail-ie0-f179.google.com with SMTP id qd14so197166ieb.38 for ; Thu, 11 Apr 2013 21:57:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=CXG178uE+B/kF6ww0hqQ1Vn1fVn8WwKu2nO7pTU4K0A=; b=PHAYexHfatELeDXqTnOGT2hHZEtATPDQRcMSfWhiJw45MkfAmU7E6FTMtMK9cgGhA3 lCJ9R/2uf4oh85deZMfJP9p9P8K0VX3isg2JXwY6O4zDRn/h8mVXxxeAjjWBStuyprFZ yE8iRLliyKyMuIfB1I6AglHCAWAAkh5mEJwrq3PkvoXsK+c9Ho0DunOWWXWZKfTHxqiQ tqfxXfPJzdh9ge/yG0n3onj5K2K2E5kFIyJjGaOt/6mVApMfEJLPXzxnzD6wQp4P7nqe iufPfocscnlEX0sSdXdYFAkj5xbVnibJN0O7BiJoI4E/aIPURGvr/K5Cly+PONH6Oy2H y3dQ== MIME-Version: 1.0 X-Received: by 10.50.12.229 with SMTP id b5mr650535igc.105.1365742643392; Thu, 11 Apr 2013 21:57:23 -0700 (PDT) Received: by 10.231.146.1 with HTTP; Thu, 11 Apr 2013 21:57:23 -0700 (PDT) In-Reply-To: <1365740112.75877.YahooMailNeo@web190702.mail.sg3.yahoo.com> References: <1364377874.13753.YahooMailNeo@web194703.mail.sg3.yahoo.com> <1364577771.12724.YahooMailNeo@web194704.mail.sg3.yahoo.com> <1364719534.91394.YahooMailNeo@web194703.mail.sg3.yahoo.com> <1365042870.89547.YahooMailNeo@web194702.mail.sg3.yahoo.com> <1365740112.75877.YahooMailNeo@web190702.mail.sg3.yahoo.com> Date: Thu, 11 Apr 2013 23:57:23 -0500 Message-ID: Subject: Re: Reduce starts before map completes (at 23%) From: shanthi k To: user@hadoop.apache.org, Sai Sai Content-Type: multipart/alternative; boundary=14dae9340b491d375804da22bd57 X-Virus-Checked: Checked by ClamAV on apache.org --14dae9340b491d375804da22bd57 Content-Type: text/plain; charset=ISO-8859-1 hello Sai, As you said Reducer starts only after completion of mapper....... U r doubting that y reducer is staring at 23% of mapper right? The answer is,there is a copy phase between mapper and reducer......after mapper as completed some tasks ,then those tasks are being copied to reducer......by copying also you can see that reducer % is increasing.......you also notice that only after mapper has completed 100% reducer completes its job 100%........thanq,Hope u will understand On Thu, Apr 11, 2013 at 11:15 PM, Sai Sai wrote: > I am running the wordcount from hadoop-examples, i am giving as input a > bunch of test files, i have noticed in the output given below reduce starts > when the map is at 23%, i was wondering if it is not right that reducers > will start only after the complete mapping is done which mean when map is > 100% then i thought the reducers will start. Why r the reducers starting > when map is still at 23%. > > 13/04/11 21:10:32 INFO mapred.JobClient: map 0% reduce 0% > 13/04/11 21:10:56 INFO mapred.JobClient: map 1% reduce 0% > 13/04/11 21:10:59 INFO mapred.JobClient: map 2% reduce 0% > 13/04/11 21:11:02 INFO mapred.JobClient: map 3% reduce 0% > 13/04/11 21:11:05 INFO mapred.JobClient: map 4% reduce 0% > 13/04/11 21:11:08 INFO mapred.JobClient: map 6% reduce 0% > 13/04/11 21:11:11 INFO mapred.JobClient: map 7% reduce 0% > 13/04/11 21:11:17 INFO mapred.JobClient: map 8% reduce 0% > 13/04/11 21:11:23 INFO mapred.JobClient: map 10% reduce 0% > 13/04/11 21:11:26 INFO mapred.JobClient: map 12% reduce 0% > 13/04/11 21:11:32 INFO mapred.JobClient: map 14% reduce 0% > 13/04/11 21:11:44 INFO mapred.JobClient: map 23% reduce 0% > 13/04/11 21:11:50 INFO mapred.JobClient: map 23% reduce 1% > 13/04/11 21:11:53 INFO mapred.JobClient: map 33% reduce 7% > 13/04/11 21:12:02 INFO mapred.JobClient: map 42% reduce 7% > > Please pour some light. > Thanks > Sai > --14dae9340b491d375804da22bd57 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
hello Sai,

As you said Reducer starts on= ly after completion of mapper....... U r doubting that y reducer is staring= at 23% of mapper right? The answer is,there is a copy phase between mapper= and reducer......after mapper as completed some tasks ,then those tasks ar= e being copied to reducer......by copying also you can see that reducer % i= s increasing.......you also notice that only after mapper has completed 100= % reducer completes its job 100%........thanq,Hope u will understand


On Thu,= Apr 11, 2013 at 11:15 PM, Sai Sai <saigraph@yahoo.in> wrote= :
I am running the wordcount from hadoop-exa= mples, i am giving as input a bunch of test files, i have noticed in the ou= tput given below reduce starts when the map is at 23%, i was wondering if i= t is not right that reducers will start only after the complete mapping is = done which mean when map is 100% then i thought the reducers will start. Wh= y r the reducers starting when map is still at 23%.

13/04/11 21:10:32 INFO mapre= d.JobClient: =A0map 0% reduce 0%
13/04/11 21:10:56 INFO mapred.J= obClient: =A0map 1% reduce 0%
13/04/11 21:= 10:59 INFO mapred.JobClient: =A0map 2% reduce 0%
13/04/11 21:11:02 INFO mapred.JobClient: =A0map 3% r= educe 0%
13/04/11 21:11:05 INFO mapred.J= obClient: =A0map 4% reduce 0%
13/04/11 21:11:08 INFO mapred.JobClient: =A0map 6% reduce 0%
13/04/11 21:11:11 INFO mapred.JobClient: =A0map 7% reduce 0%
13/04/11 21:11:17 INFO mapred.JobClient:= =A0map 8% reduce 0%
13/04= /11 21:11:23 INFO mapred.JobClient: =A0map 10% reduce 0%
13/04/11 21:11:26 INFO mapred.J= obClient: =A0map 12% reduce 0%
13/04/11 21:11:32 INFO mapred.JobClient: =A0map 14% reduce 0%
13/04/11 21:11:44 INFO mapred.JobClient: =A0map 23% reduce 0%
13/04/11 21:11:50 INFO mapred.JobClient= : =A0map 23% reduce 1%
13/= 04/11 21:11:53 INFO mapred.JobClient: =A0map 33% reduce 7%
13/04/11 21:12:02 INFO mapred.J= obClient: =A0map 42% reduce 7%

Please pour some light.
Thanks
Sai

--14dae9340b491d375804da22bd57--