hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tarandeep Singh <tarand...@gmail.com>
Subject Hadoop not utilizing max reducer capacity + reducer stuck in pending state
Date Wed, 18 Aug 2010 21:44:04 GMT

I am seeing some strange behavior in Hadoop - I am running a small test
cluster with a capacity of 18 mappers and 18 reducers. I fire a lot of jobs
simultaneously and over time I have observed Hadoop is not utilizing all the
18 slots for the reducers.

And now even if I run just one job (no other jobs running), it starts less
than 18 reducers. Initially it was starting all 18 but gradually it
decreased. For example it started only 13 reducers for a job that I just

Further, one reducer is stuck in pending state for a very long time. While
all other reducers finished, one reducer was stuck in pending state for at
least 20-30 minutes.

The mappers seem to be doing fine. Any thoughts/suggestions what could be
happening here?

Cluster conf-
1) Master- also runs 4 mappers + 4 reducers
2) 2 slaves- run 7 mappers + 7 reducers

I run ganglia monitoring system and I can tell you system was not overloaded
at any time.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message