hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ger...@dds.nl
Subject Re: Slow reduce task
Date Mon, 01 Oct 2007 13:27:00 GMT
I am experiencing a similar problem (hadoop 0.14.1 on a 2-node pool  
running java 1.5.0). Tried both grep and pi examples. Map tasks  
completed 100%, reduce tasks hang at 16%:

   INFO mapred.JobClient: map 100% reduce 16%

Job details view on http://localhost:50070jobdetails.jsp?jobid=... etc  
shows no Failed/Killed Task Attempts.

A tail on the tasktracker log keeps reiterating:

   ... INFO org.apache.hadoop.mapred.TaskTracker: task_... 0.25%  
reduce > copy (3 of 4 at 0.00 MB/s)

Ger-Jan

Quoting Ming Yang <minghsien@gmail.com>:

> Hi,
>
> I am using hadoop 0.14.1 on Ubuntu 7.04 (kernel version 2.6.20)
> The Java version is 1.5.0.12. There are no failed tasks and no
> lost task trackers.. What I observed is the machine only finished
> part of the reduce tasks and became idle. Could the issue come
> from my HDFS since the status showed the transfer rate is so low?
>
> Thanks,
>
> Ming
>
>
> 2007/9/30, Arun C Murthy <arunc@yahoo-inc.com>:
>> Ming Yang,
>>
>> On Sun, Sep 30, 2007 at 01:13:07PM -0400, Ming Yang wrote:
>> >Hi,
>> >
>> >I set up a 2-node Hadoop cluster, whose nodes are all in
>> >the same network and ran the 'grep' example. The map tasks
>> >were distributed among the two machines and ran without any
>> >problem. However, the reduce task, which is running at the slave
>> >node, doesn't seem to finish and stops at 11%. I checked the
>> >reduce task tracker and it shows the following message:
>> >
>> >reduce > copy (5 of 15 at 0.00 MB/s) >
>> >
>> >Can anyone let me know where the problem comes from?
>> >and how to fix it? I really appreciate it!
>> >
>>
>> Could you provide details on the hadoop version, platform etc.?   
>> Were there any failed tasks, lost task-trackers?
>>
>> Arun
>>
>> >Thank you,
>> >
>> >Ming Yang
>>
>



-- 
"The only truly next-gen console out there is the Wii. Everything else  
is just a
video card and processor upgrade."

A.J. McGee, http://www.computerandvideogames.com/article.php?id=144515

Mime
View raw message