hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From JAX <jayunit...@gmail.com>
Subject Re: A small portion of map tasks slows down the job
Date Wed, 03 Oct 2012 01:23:42 GMT
This is reasonable if you have any kind of trends  in the ordering of your data or  any computation
in the mappers.

You can use a smaller input split to
Reduce the load on each individual mapper so that large blocks of records that take a long
time To Process are less likely to clog one mapper.

Jay Vyas 

On Oct 2, 2012, at 9:04 PM, Huanchen Zhang <iamzhanghc@gmail.com> wrote:

> Hello,
> I have a small portion of map tasks whose output is much larger than others (more spills).
So the reducer is mainly waiting for these a few map tasks. Is there a good solution for this
problem ?
> Thank you.
> Best,
> Huanchen

View raw message