hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abdul Navaz <navaz....@gmail.com>
Subject Hadoop shuffling traffic
Date Fri, 26 Sep 2014 00:36:46 GMT

I am having a Hadoop cluster with 1 name node and 3 data nodes. I running
sample word count job on 1GB of file which is distributed among the HDFS.

When I run the map reduce job, before even completing the mapping 100 %
reduce starts.  Say for eg map 40% reduce 10% etc.

I would like to know when the shuffling traffic starts ?

->  Is there any way to find out when exactly shuffling started ?  Does it
generate any syslog in the logs .
-> How to find the total amount of shuffling traffic?

Thanks & Regards,

Abdul Navaz
Research Assistant
University of Houston Main Campus, Houston TX
Ph: 281-685-0388

View raw message