hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Why does the default HEARTBEAT_INTERVAL value is 3?
Date Tue, 09 Feb 2010 19:29:47 GMT
On a small cluster, I'm of the opinion that a value less than 3 would
actually be useful in reducing job startup time a little bit.

https://issues.apache.org/jira/browse/MAPREDUCE-1266

The issue got stalled a bit. If you want it, pipe up on the JIRA :)
Especially if you have hard data indicating this is a good idea (I
never had the time to really prove it)

-Todd

On Tue, Feb 9, 2010 at 10:21 AM, E. Sammer <eric@lifeless.net> wrote:
> On 2/9/10 11:52 AM, ChingShen wrote:
>>
>> Hi,
>>
>>  I have a question about HEARTBEAT_INTERVAL.
>>  Why does the default HEARTBEAT_INTERVAL value is 3 rather than 2 or 1?
>> any
>> resources?
>
> Shen:
>
> While I don't have a good answer for why the number 3 was chosen (actually,
> I think it's 5 seconds on heartbeats and the 3 seconds is how often a task
> tracker thread checks if progress is being made or something like that), I
> can tell you that there's network chatter caused by the heartbeat. You
> wouldn't want heartbeat to be any faster as you would unnecessarily cause
> network congestion and force the job tracker to do additional (possibly
> unnecessary) work. As the cluster grows, the heartbeat interval is increased
> leading to even less frequent check-ins to attempt to mitigate the
> congestion / high concurrency on the JT.
>
> One of the down sides to this is that tasks aren't given to task trackers as
> quickly as they could be, but there are probably better ways of decreasing
> the amount of time required to hand out work rather than simply increasing
> the heartbeat rate.
>
> Keep in mind that most Hadoop jobs run for long periods of time, so the
> slight delay in handing out tasks isn't a huge problem and 3 to 5 seconds is
> more than sufficient to know that a task tracker is alive and healthy.
>
> Hope this helps.
> --
> Eric Sammer
> eric@lifeless.net
> http://esammer.blogspot.com
>

Mime
View raw message