hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bae, Jae Hyeon" <metac...@gmail.com>
Subject Insufficient reduce task capacity
Date Thu, 27 Aug 2009 03:33:41 GMT
Hi, My name is Jay from Korea.

While running Hadoop cluster with version 0.18.3, I added several
machines to the cluster a few days ago.

Legacy machines have 2 CPU and 8 GB memories, I set
mapred.tasktracker.map.tasks.maximum and
mapred.tasktracker.reduce.tasks.maximun as 2.

New machines attached have 4 CPU and 16 GB memories, I set
mapred.tasktracker.map.tasks.maximum and
mapred.tasktracker.reduce.tasks.maximun as 4.

It leads to increase map task capacity and reduce task capacity. I
executed heavy job which could take up all resources of Hadoop
cluster, I was surprised it could not take up all resources. It was
using every slot of map task but its several reduce works were
pending. I checked the task trackers of Hadoop Machine List page, I
finally discovered that 4 CPU machines were running only 3 reducers.

I also checked logs of task tracker in 4 CPU machines, but I didn't
find anything wrong yet, except LOW ON THREAD and OUT OF THREAD from
jetty web server. I don't think it affected insufficient reduce task
capacity.

Do you know how to solve this problem?

Regards Jay

Mime
View raw message