hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod K V (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1266) Allow heartbeat interval smaller than 3 seconds for tiny clusters
Date Tue, 08 Dec 2009 04:05:18 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787295#action_12787295
] 

Vinod K V commented on MAPREDUCE-1266:
--------------------------------------

bq. BTW, the inspiration was this blog post: http://pero.blogs.aprilmayjune.org/2009/11/30/improve-performance-on-small-hadoop-clusters/
Just a thought, is this improvement perceivable at full/heavy job-load? Heart-beats in an
idle cluster are mostly very cheap. Heavy load involves costlier heartbeat processing and
so may or may not show the performance improvements you have outlined..

> Allow heartbeat interval smaller than 3 seconds for tiny clusters
> -----------------------------------------------------------------
>
>                 Key: MAPREDUCE-1266
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1266
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker, task, tasktracker
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Priority: Minor
>
> For small clusters, the heartbeat interval has a large effect on job latency. This is
especially true on pseudo-distributed or other "tiny" (<5 nodes) clusters. It's not a big
deal for production, but new users would have a happier first experience if Hadoop seemed
snappier.
> I'd like to change the minimum heartbeat interval from 3.0 seconds to perhaps 0.5 seconds
(but have it governed by an undocumented config parameter in case people don't like this change).
The cluster size-based ramp up of interval will maintain the current scalable behavior for
large clusters with no negative effect.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message