hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6640) AM heartbeat stuck when responseId overflows MAX_INT
Date Tue, 22 Aug 2017 22:00:07 GMT

    [ https://issues.apache.org/jira/browse/YARN-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16137475#comment-16137475

Wangda Tan commented on YARN-6640:

Thanks [~botong] for reporting and working on this case. I think this is pretty severe issue
which we should fix and backport to previous releases. Marked as blocker for all > 2.8

We have two choices, one is change type of response id from int to long. I personally don't
prefer that because even long could be exhausted if they have an app runs for hundred years
:),  and it has compatibility issue as well.

I prefer to reuse int like what you did in the patch, if value equals MAX_INT, we will set
it back to 0 and handle the special checking logic. I don't suggest have a special reserved-id,
this makes code become confusing.

Another potential problem is, we only fail when request.responseId < lastResponse.responseId
- 1, I think we should also fail when {{request.responseId > lastResponseId}}.


+ [~jlowe]. 

>  AM heartbeat stuck when responseId overflows MAX_INT
> -----------------------------------------------------
>                 Key: YARN-6640
>                 URL: https://issues.apache.org/jira/browse/YARN-6640
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Botong Huang
>            Assignee: Botong Huang
>            Priority: Blocker
>             Fix For: 2.9.0, 3.0.0-beta1, 2.8.2
>         Attachments: YARN-6640.v1.patch
> The current code in {{ApplicationMasterService}}: 
> if ((request.getResponseId() + 1) == lastResponse.getResponseId()) {/* old heartbeat
*/  return lastResponse;}
> else if (request.getResponseId() + 1 < lastResponse.getResponseId()) { throw ... }
> process the heartbeat...
> When a heartbeat comes in, in usual case we are expecting request.getResponseId() ==
lastResponse.getResponseId(). The “if“ is for the duplicate heartbeat that’s one step
old, the “else if” is to throw and complain for heartbeats more than two steps old, otherwise
we accept the new heartbeat and process it.
> So the bug is: when lastResponse.getResponseId() == MAX_INT, the newest heartbeat comes
in with responseId == MAX_INT. However reponseId + 1 will be MIN_INT, and we will fall into
the “else if” case and RM will throw. Then we are stuck here…

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message