hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tsuyoshi OZAWA <ozawa.tsuyo...@lab.ntt.co.jp>
Subject Re: How to reduce total shuffle time
Date Tue, 28 Aug 2012 08:37:42 GMT
It depends of workload. Could you tell us more specification about
your job? In general case which reducers are bottleneck, there are
some tuning techniques as follows:
1. Allocate more memory to reducers. It decreases disk IO of reducers
when merging and running reduce functions.
2. Use combine function, which enable mapper-side aggregation
processing, if your MR job consists of the operations that satisfy
both the commutative and the associative low.

See also about combine functions:
http://wiki.apache.org/hadoop/HadoopMapReduce

Tsuyoshi

On Tuesday, August 28, 2012, Gaurav Dasgupta wrote:
>
> Hi,
>
> I have run some large and small jobs and calculated the Total Shuffle Time for the jobs.
I can see that the Total Shuffle Time is almost half the Total Time which was taken by the
full job to complete.
>
> My question, here, is that how can we decrease the Total Shuffle Time? And doing so,
what will be its effect on the Job?
>
> Thanks,
> Gaurav Dasgupta

Mime
View raw message