hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abdul Navaz <navaz....@gmail.com>
Subject Re: Hadoop shuffling traffic
Date Thu, 02 Oct 2014 18:03:25 GMT
Hello Pramod,

This is great work !. Thank you for sharing the report.

Thanks & Regards,

Abdul Navaz
Research Assistant
University of Houston Main Campus, Houston TX
Ph: 281-685-0388


From:  Pramod Biligiri <pramodbiligiri@gmail.com>
Reply-To:  <user@hadoop.apache.org>
Date:  Thursday, October 2, 2014 at 12:44 AM
To:  "zookeeper-user@hadoop.apache.org" <user@hadoop.apache.org>
Subject:  Re: Hadoop shuffling traffic

Hi Abdul,
That is the right metric. You can take a look at this report we made on this
earlier: 
http://www.slideshare.net/pramodbiligiri/shuffle-phase-as-the-bottleneck-in-
hadoop-terasort

Pramod

On Wed, Oct 1, 2014 at 6:06 PM, Abdul Navaz <navaz.enc@gmail.com> wrote:
> Hello,
> 
> This is the portion of the output which is displayed on the console when I run
> sample word count job.
> 
> map 0% reduce 0%
> 
> 14/10/01 18:37:52 INFO mapred.JobClient:  map 100% reduce 0%
> 
> 14/10/01 18:38:10 INFO mapred.JobClient:  map 100% reduce 100%
> 
> 14/10/01 18:38:12 INFO mapred.JobClient: Job complete: job_201409262002_0003
> 
> 14/10/01 18:38:12 INFO mapred.JobClient: Counters: 29
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:   Job Counters
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:     Launched reduce tasks=1
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=23511
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:     Total time spent by all reduces
> waiting after reserving slots (ms)=0
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:     Total time spent by all maps
> waiting after reserving slots (ms)=0
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:     Launched map tasks=1
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:     Data-local map tasks=1
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=14193
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:   File Output Format Counters
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:     Bytes Written=1106
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:   FileSystemCounters
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:     FILE_BYTES_READ=3059
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:     HDFS_BYTES_READ=1601
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=108400
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=1106
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:   File Input Format Counters
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:     Bytes Read=1486
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:   Map-Reduce Framework
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:     Map output materialized
> bytes=3059
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:     Map input records=6
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:     Reduce shuffle bytes=3059
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:     Spilled Records=544
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:     Map output bytes=2509
> 
> 14/10/01 18:38:12 INFO mapred.JobClient:     Total committed heap usage
> 
> 
> 
> I am trying to find the shuffling traffic that is total traffic generated when
> mappers exchange their key values pair with the reducer. Is the highlighted
> portion gives the shuffling traffic ?
> 
> 
> Thanks & Regards,
> 
> Abdul Navaz
> Research Assistant
> University of Houston Main Campus, Houston TX
> Ph: 281-685-0388 <tel:281-685-0388>
> 
> 
> 
> 
> On 9/26/14, 12:00 AM, "karthikeyan S" <karthispeaks@gmail.com> wrote:
> 
>> The reducer starts as soon as it has data available from any one of the
>> mappers.
>> The reducer keeps polling the AM and asks if any mapper has completed
>> processing. If so it fetches data from that mapper.
>> So it's not necessary for all the mappers of a task to complete for
>> the reducer to start processing.
>> 
>> When the reducers starts fetching the data from the mappers it prints
>> that info in its syslog, from what I have seen.
>> 
>> Thanks,
>> Karthik
>> 
>> On Thu, Sep 25, 2014 at 8:27 PM, Bing Jiang <jiangbinglover@gmail.com> wrote:
>>>  see mapreduce.job.reduce.slowstart.completedmaps
>>>  It gives hint of  when reduce tasks could kick off.
>>> 
>>>  2014-09-26 8:36 GMT+08:00 Abdul Navaz <navaz.enc@gmail.com>:
>>>> 
>>>>  Hello,
>>>> 
>>>>  I am having a Hadoop cluster with 1 name node and 3 data nodes. I running
>>>>  sample word count job on 1GB of file which is distributed among the HDFS.
>>>> 
>>>>  When I run the map reduce job, before even completing the mapping 100 %
>>>>  reduce starts.  Say for eg map 40% reduce 10% etc.
>>>> 
>>>>  I would like to know when the shuffling traffic starts ?
>>>> 
>>>>  ->  Is there any way to find out when exactly shuffling started ?  Does
it
>>>>  generate any syslog in the logs .
>>>>  -> How to find the total amount of shuffling traffic?
>>>> 
>>>> 
>>>> 
>>>>  Thanks & Regards,
>>>> 
>>>>  Abdul Navaz
>>>>  Research Assistant
>>>>  University of Houston Main Campus, Houston TX
>>>>  Ph: 281-685-0388 <tel:281-685-0388>
>>>> 
>>> 
>>> 
>>> 
>>>  --
>>>  Bing Jiang
>>>  Tel:(86)134-2619-1361
>>>  weibo: http://weibo.com/jiangbinglover
>>>  BLOG: www.binospace.com <http://www.binospace.com>
>>>  BLOG: http://blog.sina.com.cn/jiangbinglover
>>>  Focus on distributed computing, HDFS/HBase
>> 




Mime
View raw message