hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From abhiTowson cal <abhishek.dod...@gmail.com>
Subject Re: How to reduce total shuffle time
Date Tue, 28 Aug 2012 19:27:13 GMT
hi Gaurav,

Can you tell me how did calculated total shuffle time ?.Apart from
combiners and compression, you can also use some shuffle-sort
parameters that might increase the performance, i am not sure exactly
which parameters to tweak .Please share if you come across some other
techniques , i am very much interested.

Regards
Abhi

On Tue, Aug 28, 2012 at 3:16 AM, Gaurav Dasgupta <gdsayshi@gmail.com> wrote:
> Hi,
>
> I have run some large and small jobs and calculated the Total Shuffle Time
> for the jobs. I can see that the Total Shuffle Time is almost half the Total
> Time which was taken by the full job to complete.
>
> My question, here, is that how can we decrease the Total Shuffle Time? And
> doing so, what will be its effect on the Job?
>
> Thanks,
> Gaurav Dasgupta

Mime
View raw message