hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Botong Huang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-7102) NM heartbeat stuck when responseId overflows MAX_INT
Date Thu, 21 Sep 2017 17:20:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175133#comment-16175133
] 

Botong Huang edited comment on YARN-7102 at 9/21/17 5:19 PM:
-------------------------------------------------------------

Thanks [~jlowe] for the review and good point. We don't want RM to resync all nodes if RM
becomes slow. 

How about we take one step back, allowing {{request.responseId > lastResponseId}} as it
is now? We simply fix the overflow problem without changing anything else. Specifically, add
one check: if {{request.responseId == lastResponseId}} then skip other checks. This would
be my initial proposal in YARN-6640 v1 patch: 

{code}
	if (request.getResponseId() != lastResponse.getResponseId()) {
	    if ((request.getResponseId() + 1) == lastResponse.getResponseId()) {
	          /* heartbeat one step old, simply return lastReponse */
	          return lastResponse;
	    } else if (request.getResponseId() + 1 < lastResponse.getResponseId()) {
	          (resync NM...)
	    }
	}
       (process the heartbeat...)
{code}

There's still potential for the RM too slow causing NM resync, but only possible for the NMs
whose reponseId just wrapped around. This should be fine I guess. 


was (Author: botong):
Thanks [~jlowe] for the review and good point. We don't want RM to resync all nodes if RM
becomes slow. 

How about we take one step back, allowing {{request.responseId > lastResponseId}} as it
is now? We simply fix the overflow problem without changing anything else. Specifically, add
one check: if {{request.responseId == lastResponseId}} then skip other checks. This would
be my initial proposal in YARN-6640 v1 patch: 

{code}
	if (request.getResponseId() != lastResponse.getResponseId()) {
	    if ((request.getResponseId() + 1) == lastResponse.getResponseId()) {
	          /* heartbeat one step old, simply return lastReponse */
	          return lastResponse;
	    } else if (request.getResponseId() + 1 < lastResponse.getResponseId()) {
	          (resync NM)
	    }
	}
       (process the heartbeat...)
{code}

There's still potential for the RM too slow causing NM resync, but only possible for the NMs
whose reponseId just wrapped around. This should be fine I guess. 

> NM heartbeat stuck when responseId overflows MAX_INT
> ----------------------------------------------------
>
>                 Key: YARN-7102
>                 URL: https://issues.apache.org/jira/browse/YARN-7102
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Botong Huang
>            Assignee: Botong Huang
>            Priority: Critical
>         Attachments: YARN-7102.v1.patch, YARN-7102.v2.patch, YARN-7102.v3.patch, YARN-7102.v4.patch,
YARN-7102.v5.patch, YARN-7102.v6.patch
>
>
> ResponseId overflow problem in NM-RM heartbeat. This is same as AM-RM heartbeat in YARN-6640,
please refer to YARN-6640 for details. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message