hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@hortonworks.com>
Subject Re: Performance of mappers
Date Fri, 05 Aug 2011 18:05:52 GMT
Which release of Hadoop are you running?

What do the logs on the TaskTracker tell you during the time the slow tasks are getting launched?

hadoop-0.20.203 has a ton of bug fixes since hadoop-0.20.2 which help fix issues with slow
launches - you might want to upgrade.

Arun

On Aug 5, 2011, at 11:02 AM, Iman E wrote:

> Hello all,
> I have a question regarding the mappers. I can see from the logs that the start time
of the mapper is different from start time of logging. I am having a problem because that
time difference sometimes is few seconds, but other times it is
>  
> For example, one mapper that is supposed to read 65 MB. Its start time is 12:30:53 whereis
the logging start time is 12:33:01 and the end time is 12:33:20. All the laoded data are local
to the same rack.
> In a perfect run, these numbers are as follows: the start time is 18:15:45, logging start
time: 18:15:48, and end time: 18:16:02.
>  
>  
> I am running a job of more than 2400 mapper. Because of the above problem, instead of
the job taking 15-20 mins  on 14 machine ( it happened in few runs), other times it is taking
more than 70 mins. Any suggestions how to fix this problem or what could possibly be causing
it.
>  
> Thanks,
> Iman


Mime
View raw message