Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CFAB617CA0 for ; Mon, 6 Oct 2014 19:19:35 +0000 (UTC) Received: (qmail 891 invoked by uid 500); 6 Oct 2014 19:19:26 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 777 invoked by uid 500); 6 Oct 2014 19:19:26 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 767 invoked by uid 99); 6 Oct 2014 19:19:26 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Oct 2014 19:19:26 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=HTML_FONT_FACE_BAD,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of renato.moutinho@gmail.com designates 74.125.82.48 as permitted sender) Received: from [74.125.82.48] (HELO mail-wg0-f48.google.com) (74.125.82.48) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Oct 2014 19:19:00 +0000 Received: by mail-wg0-f48.google.com with SMTP id k14so6105356wgh.31 for ; Mon, 06 Oct 2014 12:18:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=2GocN4QWrqxb4Jpk+jrx6EctnIUfVsL0CZHab9mZMX8=; b=rPdxO80rwBXEtUQkBaSRX2JAp9ElZheDdMpgX7gBbY/aBXldhkPnEKVDRsAVthkU7E 79msMFSe/lzc0nBjV8z79k1zXLoyeamvDRCP8QFNm77wsug+B25IbqCYLQbuU9nT5837 uOp6k8iPgs2hAEoRq5eMd9p+0dmsNYewf4NpLLG309HPCMXzBC/KV9l72CXymduiBEpF XIxfejhzXhudA+KL7+4KW6dBkS6ybF+ZKMP3oMsSYZnEY/CAqY3FnHXlBbTXj8bRxhpR hDChWprQ8oS2JDcokjQ3QSftRCSviQEQVB2nhPH1QFs4zRCrE/vSAHF+nz8hCWm93ERa vmSA== MIME-Version: 1.0 X-Received: by 10.180.91.199 with SMTP id cg7mr21670402wib.10.1412623139718; Mon, 06 Oct 2014 12:18:59 -0700 (PDT) Received: by 10.194.174.39 with HTTP; Mon, 6 Oct 2014 12:18:59 -0700 (PDT) In-Reply-To: <5431BDBE.6050003@ulul.org> References: <-4690291296535386739@unknownmsgid> <5431BDBE.6050003@ulul.org> Date: Mon, 6 Oct 2014 16:18:59 -0300 Message-ID: Subject: Re: Reduce phase of wordcount From: Renato Moutinho To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=f46d043bdeee71ca310504c5f45b X-Virus-Checked: Checked by ClamAV on apache.org --f46d043bdeee71ca310504c5f45b Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi folks, just as a feeback: increasing mapred.tasktracker.reduce.tasks.maximum had no effect (it was already set to 8) and the job created only 1 reducer (my original scenario). However, adding mapred.reduce.tasks and setting to some higher than 1 value (I=C2=B4ve set to 7) made hadoop spawn that much r= educe tasks (seven on my example) and the execution time went down to around 29 minutes (also, my servers are now frying cpu....) ! My next step (I=C2=B4m pushing it to the maximum) is adding a combiner.. And no... I haven=C2=B4t setup this cluster just for running wordcount.Kkkkkkkk.... I'm still getting to know hadoop. :-) Thanks a lot for your help ! Regards, Renato Moutinho 2014-10-05 18:53 GMT-03:00 Ulul : > Hi > > You indicate that you have just one reducer, which is the default in > Hadoop 1 but quite insufficient for a 7 slave nodes cluster. > You should increase mapred.reduce.tasks use combiners and maybe tune > mapred.reduce.tasktracker.reduce.tasks.maximum > > Hope that helps > Ulul > > Le 05/10/2014 16:53, Renato Moutinho a =C3=A9crit : > > Hi there, > > thanks a lot for taking the time to answer me ! Actually, this > "issue" happens after all the map tasks have completed (I'm looking at th= e > web interface). I'll try to diagnose if it's an issue with the number of > threads.. I suppose I'll have to change the logging configuration to find > what's going on.. > > The only that's getting to me is the fact that the lines are repeated on > the log.. > > Regards, > > Renato Moutinho > > > > Em 05/10/2014, =C3=A0s 10:52, java8964 escreveu: > > Don't be confused by 6.03 MB/s. > > The relationship between mapper and reducer is M to N relationship, > which means the mapper could send its data to all reducers, and one reduc= er > could receive its input from all mappers. > > There could be a lot of reasons why you think the reduce copying phase > is too slow. It could be the mappers are still running, there is no data > generated for reducer to copy yet; or there is no enough threads in eithe= r > mapper or reducer to utilize remaining cpu/memory/network bandwidth. You > can google the hadoop configurations to adjust them. > > But just because you can get 60M/s in scp, then complain only getting > 6M/s in the log is not fair to hadoop. You one reducer needs to copy data > from all the mappers, concurrently, makes it impossible to reach the same > speed as one to one point network transfer speed. > > The reducer stage is normally longer than map stage, as data HAS to be > transferred through network. > > But in word count example, the data needs to be transferred should be > very small. You can ask the following question by yourself: > > 1) Should I use combiner in this case? (Yes, for word count, it reduces > the data needs to be transferred). > 2) Do I use all the reducers I can use, if my cluster is under utilized > and I want my job to finish fast? > 3) Can I add more threads in the task tracker to help? You need to dig > into your log to find out if your mapper or reducer are waiting for the > thread from thread pool. > > Yong > > ------------------------------ > Date: Fri, 3 Oct 2014 18:40:16 -0300 > Subject: Reduce phase of wordcount > From: renato.moutinho@gmail.com > To: user@hadoop.apache.org > > Hi people, > > I=C2=B4m doing some experiments with hadoop 1.2.1 running the wordco= unt > sample on an 8 nodes cluster (master + 7 slaves). Tuning the tasks > configuration I=C2=B4ve been able to make the map phase run on 22 minutes= .. > However the reduce phase (which consists of a single job) stucks at some > points making the whole job take more than 40 minutes. Looking at the log= s, > I=C2=B4ve seen several lines stuck at copy on different moments, like thi= s: > > 2014-10-03 18:26:34,717 INFO org.apache.hadoop.mapred.TaskTracker: > attempt_201408281149_0019_r_000000_0 0.3302721% reduce > copy (971 of 980 > at 6.03 MB/s) > > 2014-10-03 18:26:37,736 INFO org.apache.hadoop.mapred.TaskTracker: > attempt_201408281149_0019_r_000000_0 0.3302721% reduce > copy (971 of 980 > at 6.03 MB/s) > > 2014-10-03 18:26:40,754 INFO org.apache.hadoop.mapred.TaskTracker: > attempt_201408281149_0019_r_000000_0 0.3302721% reduce > copy (971 of 980 > at 6.03 MB/s) > > 2014-10-03 18:26:43,772 INFO org.apache.hadoop.mapred.TaskTracker: > attempt_201408281149_0019_r_000000_0 0.3302721% reduce > copy (971 of 980 > at 6.03 MB/s) > > > Eventually the job end, but this information, being repeated, makes me > think it=C2=B4s having difficulty transferring the parts from the map nod= es. Is > my interpretation correct on this ? The trasnfer rate is waaay too slow i= f > compared to scp file transfer between the hosts (10 times slower). Any > takes on why ? > > Regards, > > Renato Moutinho > > > --f46d043bdeee71ca310504c5f45b Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi folks,

=C2=A0=C2=A0=C2=A0 just a= s a feeback: increasing mapred.tasktracker.reduce.tasks.maximum had no effe= ct (it was already set to 8) and the job created only 1 reducer (my origina= l scenario). However, adding mapred.reduce.tasks and setting to some higher= than 1 value (I=C2=B4ve set to 7) made hadoop spawn that much reduce tasks= (seven on my example) and the execution time went down to around 29 minute= s (also, my servers are now frying cpu....) ! My next step (I=C2=B4m pushin= g it to the maximum) is adding a combiner..

And no... I h= aven=C2=B4t setup this cluster just for running wordcount.Kkkkkkkk.... I= 9;m still getting to know hadoop. :-)

Thanks a lot for your he= lp !

Regards,

Renato Moutinho
<= br>
2014-10-05 18:53 GMT-03:00 Ulul <hadoop@ulul.o= rg>:
=20 =20 =20
Hi

You indicate that you have just one reducer, which is the default in Hadoop 1 but quite insufficient for a 7 slave nodes cluster.
You should increase mapred.reduce.tasks use combiners and maybe tune mapred.reduce.tasktracker= .reduce.tasks.maximum

Hope that helps
Ulul
Le 05/10/2014 16:53, Renato Moutinho a =C3=A9crit=C2=A0:
=20
Hi there,

=C2=A0 =C2=A0 =C2=A0thanks a lot for taking the time to answer m= e ! Actually, this "issue" happens after all the map tasks ha= ve completed (I'm looking at the web interface). I'll try to diagnose if it's an issue with the number of threads.. I suppos= e I'll have to change the logging configuration to find what'= s going on..

The only that's getting to me is the fact that the lines are repeated on the log..

Regards,

Renato Moutinho



Em 05/10/2014, =C3=A0s 10:52, java8964 <java8964@hotmail.com> escreveu:

=20
Don't be confused by 6.03 MB/s.

The relationship between mapper and reducer is M to N relationship, which means the mapper could send its data to all reducers, and one reducer could receive its input from all mappers.

There could be a lot of reasons why you think the reduce copying phase is too slow. It could be the mappers are still running, there is no data generated for reducer to copy yet; or there is no enough threads in either mapper or reducer to utilize remaining cpu/memory/network bandwidth. You can google the hadoop configurations to adjust them.

But just because you can get 60M/s in scp, then complain only getting 6M/s in the log is not fair to hadoop. You one reducer needs to copy data from all the mappers, concurrently, makes it impossible to reach the same speed as one to one point network transfer speed.

The reducer stage is normally longer than map stage, as data HAS to be transferred through network.

But in word count example, the data needs to be transferred should be very small. You can ask the following question by yourself:

1) Should I use combiner in this case? (Yes, for word count, it reduces the data needs to be transferred).
2) Do I use all the reducers I can use, if my cluster is under utilized and I want my job to finish fast?
3) Can I add more threads in the task tracker to help? You need to dig into your log to find out if your mapper or reducer are waiting for the thread from thread pool.

Yong

Date: Fri, 3 Oct 2014 18:40:16 -0300
Subject: Reduce phase of wordcount
From: renato.moutinho@gmail.com
To: user@hadoop.apache.org

Hi people,

=C2=A0=C2=A0=C2=A0 I=C2=B4m doing some experiments wi= th hadoop 1.2.1 running the wordcount sample on an 8 nodes cluster (master + 7 slaves). Tuning the tasks configuration I=C2=B4ve been able to make the map pha= se run on 22 minutes.. However the reduce phase (which consists of a single job) stucks at some points making the whole job take more than 40 minutes. Looking at the logs, I=C2=B4ve seen several lines stuck at copy on different moments, like this:

2014-10-03 18:26:34,717 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201408281149_0019_r_000000_0 0.3302721% reduce > copy (971 of 980 at 6.03 MB/s) >
2014-10-03 18:26:37,736 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201408281149_0019_r_000000_0 0.3302721% reduce > copy (971 of 980 at 6.03 MB/s) >
2014-10-03 18:26:40,754 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201408281149_0019_r_000000_0 0.3302721% reduce > copy (971 of 980 at 6.03 MB/s) >
2014-10-03 18:26:43,772 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201408281149_0019_r_000000_0 0.3302721% reduce > copy (971 of 980 at 6.03 MB/s) >

Eventually the job end, but this information, being repeated, makes me think it=C2=B4s having difficulty transferring the parts from the map nodes. Is my interpretation correct on this ? The trasnfer rate is waaay too slow if compared to scp file transfer between the hosts (10 times slower). Any takes on why ?

Regards,

Renato Moutinho


--f46d043bdeee71ca310504c5f45b--