hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xianyin Xin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3630) YARN should suggest a heartbeat interval for applications
Date Thu, 28 May 2015 02:38:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14562211#comment-14562211

Xianyin Xin commented on YARN-3630:

Thanks for your comments, [~vvasudev]!
your patch doesn't check if the calculated interval is greater than the ping interval to determine
liveliness for the AM and the NM. Is that by design?
It's true we should do that. But in this patch I haven't add the mechanism of determining
a up limit for the {{nextHeartbeatInterval}}. I think the limit should much less than the
ping interval which is 10 minutes by default. Other hand, do you think a hard configurable
limit is accepted? 
With respect to adaptive heartbeats for the NMs - my concern is that the proposed solution
will lead to behaviour where the NMs will be told to back off - the NMs will wait for sometime
- the RM will receive a flood of NM updates - leading to the NMs being told to back off and
so on and so forth. We'll end up in a situation where the pings will become clustered around
particular time intervals, leading to container allocation and release delays. You might be
better off picking a random interval between the default interval and the calculated interval
to spread out the NM pings
Thanks for reminding, it's a situation I didn't think much. I think your suggestion is a nice

> YARN should suggest a heartbeat interval for applications
> ---------------------------------------------------------
>                 Key: YARN-3630
>                 URL: https://issues.apache.org/jira/browse/YARN-3630
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager, scheduler
>    Affects Versions: 2.7.0
>            Reporter: Zoltán Zvara
>            Assignee: Xianyin Xin
>            Priority: Minor
>         Attachments: Notes_for_adaptive_heartbeat_policy.pdf, YARN-3630.001.patch.patch,
> It seems currently applications - for example Spark - are not adaptive to RM regarding
heartbeat intervals. RM should be able to suggest a desired heartbeat interval to applications.

This message was sent by Atlassian JIRA

View raw message