hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@yahoo-inc.com>
Subject Re: map-tasks "beating up" a node
Date Thu, 13 Nov 2008 22:23:53 GMT

On Nov 13, 2008, at 1:59 PM, Brian Bockelman wrote:

> Hey all,
>
> When we run a large task (lots of intermediate output), we have a  
> real problem with it "beating up" nodes during the shuffle phase.   
> About half of our nodes are completely overwhelmed by the number of  
> reducers requesting data to be copied (somewhere between 50 and 75  
> files appear to be open); this causes wait_io on the node to go up  
> to about 50-75% of the CPU.  The number of active threads is about  
> 45.  The amount of intermediate data is about 10GB per node.
>
> Anyone else run into this problem?  Any hints? My first thought is  
> to ask the user to use a combiner so there are less files to work  
> with in the first place.
>

Combiners are useful, they help in cutting down the amount of data  
which needs to be shuffled.

Have you tried compressing map-outputs?
http://hadoop.apache.org/core/docs/current/mapred_tutorial.html#Data+Compression

A heavy-handed option is to cut down the no. of per-reducer threads  
which do the shuffle: mapred.reduce.parallel.copies. We recommend ~20,  
try with a lesser number than that...

Arun

Mime
View raw message