hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Stewart <robstewar...@googlemail.com>
Subject Slow final few reducers
Date Sat, 11 Dec 2010 11:05:02 GMT

I have a problem with a MapReduce job I am trying to run on a 32 node cluster.

The final few reducers take a *lot* longer than the rest. e.g. If I
specify 100 reducers, the first 90 will complete in 5 minutes, and
then the remaining 10 reducers might take 10 minutes.

Same is true for any number of reducers... 200 reducers: 180/190 will
complete in 5 minutes, and the last 10/20 will take 10 minutes.

Is this normal Hadoop behavior? I know that the output of the Reducer
function is not sorted, so can't figure out why this decline of
performance at the tail end of the job?


Rob Stewart

View raw message