hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Renato Moutinho <renato.mouti...@gmail.com>
Subject Reduce phase of wordcount
Date Fri, 03 Oct 2014 21:40:16 GMT
Hi people,

    I´m doing some experiments with hadoop 1.2.1 running the wordcount
sample on an 8 nodes cluster (master + 7 slaves). Tuning the tasks
configuration I´ve been able to make the map phase run on 22 minutes..
However the reduce phase (which consists of a single job) stucks at some
points making the whole job take more than 40 minutes. Looking at the logs,
I´ve seen several lines stuck at copy on different moments, like this:

2014-10-03 18:26:34,717 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201408281149_0019_r_000000_0 0.3302721% reduce > copy (971 of 980
at 6.03 MB/s) >
2014-10-03 18:26:37,736 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201408281149_0019_r_000000_0 0.3302721% reduce > copy (971 of 980
at 6.03 MB/s) >
2014-10-03 18:26:40,754 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201408281149_0019_r_000000_0 0.3302721% reduce > copy (971 of 980
at 6.03 MB/s) >
2014-10-03 18:26:43,772 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201408281149_0019_r_000000_0 0.3302721% reduce > copy (971 of 980
at 6.03 MB/s) >

Eventually the job end, but this information, being repeated, makes me
think it´s having difficulty transferring the parts from the map nodes. Is
my interpretation correct on this ? The trasnfer rate is waaay too slow if
compared to scp file transfer between the hosts (10 times slower). Any
takes on why ?

Regards,

Renato Moutinho

Mime
View raw message