hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Kumpe <christ...@kumpe.de>
Subject Re: Steps in execution time
Date Mon, 11 Apr 2011 18:34:55 GMT
Hi Abhishek,

thanks for your answer. Thus this is the reason for the 1s and 3s raster
in the whole plot.

Do you (or someone else) have any ideas what maybe is causing the few
outliers downwards?
The outliers upwards can be caused by some latencies in the network or
in the some of the nodes. No problem so far.
But are the few (but recurring) outliers downwards just a chain of
(very) lucky accidents? Or a bug in my measurement :-) At least they
fall into the 1s and 3s raster, too.
Or is there another bigger (about 6s) polling interval which is
overlaying the 1s and 3s raster.

Christian


Am Montag, den 11.04.2011, 12:38 -0400 schrieb abhishek sharma: 
> Christian,
> 
> The TaskTrackers send "heartbeat" messages to the JobTracker. The
> default interval for these messages is 3 seconds.
> This is one reason why you see the 3 second steps.
> 
> Abhishek
> 
> On Mon, Apr 11, 2011 at 3:19 AM, Christian Kumpe <christian@kumpe.de> wrote:
> > Hi,
> >
> > I'm doing some measurement on hadoop's execution time for my theses. I
> > discovered some steps in the jobs execution time when raising the
> > mappers' execution time continuously.
> > Here is a plot of the execution times with 1, 30 and 60 parallel
> > executing mappers:
> > http://baumbart.drhuim.de/executiontime.gif
> >
> > The cluster consists of 10 nodes and an overall map task capacity of 60.
> > I'm using hadoop 0.20.2.
> >
> > The many small steps about 1 second are probably caused by the polling
> > interval of the JobClient in monitorAndPrintJob(..) which is about 1
> > second.
> >
> > But can anyone give me a hint, what's causing the bigger 3 second steps?
> > I suppose it's a polling interval, too. But where? In the Jobtracker? In
> > the Tasktrackers?
> >
> > Thanks for some help and comments,
> >  Christian
> >
> >



Mime
View raw message