hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tsuyoshi OZAWA <ozawa.tsuyo...@lab.ntt.co.jp>
Subject Re: How to reduce total shuffle time
Date Tue, 28 Aug 2012 08:37:42 GMT
It depends of workload. Could you tell us more specification about
your job? In general case which reducers are bottleneck, there are
some tuning techniques as follows:
1. Allocate more memory to reducers. It decreases disk IO of reducers
when merging and running reduce functions.
2. Use combine function, which enable mapper-side aggregation
processing, if your MR job consists of the operations that satisfy
both the commutative and the associative low.

See also about combine functions:


On Tuesday, August 28, 2012, Gaurav Dasgupta wrote:
> Hi,
> I have run some large and small jobs and calculated the Total Shuffle Time for the jobs.
I can see that the Total Shuffle Time is almost half the Total Time which was taken by the
full job to complete.
> My question, here, is that how can we decrease the Total Shuffle Time? And doing so,
what will be its effect on the Job?
> Thanks,
> Gaurav Dasgupta

View raw message