hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohith Sharma K S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4497) RM might fail to restart when recovering apps whose attempts are missing
Date Wed, 13 Jan 2016 07:17:39 GMT

    [ https://issues.apache.org/jira/browse/YARN-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15095740#comment-15095740

Rohith Sharma K S commented on YARN-4497:

As a side node : since YARN-3840 removes the attempts from RMStateStore, it is very prone
to get this issue (YARN-4584) nevertheless of *without RM HA is configured and fail fast is

About the solution, it is bit tricky to identify during recovery that *whether-application-is-failed-to-store*
VS *failed-attempts-were-removed-after-interval*. So I think you can club both your solution
and [~jianhe]'s thought together, so that we can eliminate *failed-attempts-were-removed-after-interval*
attempts. And assume that attempts recovered are of failed to store only. Thoughts?
Regarding iterating appState.attempts, it can be sorted before iterating it. If attempts are
sorted, then there should not be problem with nextAttemptId.

About the patch,
# attempt.recoveredFinalStatus is being set to always to FAILED. These attempts might be KILLED/FINISHED
# *getNumFailedAppAttempts()* is violated if attempt is failed to store since this attempt
is removed from *attempts*. And also note that if attempts is failed to store, then many information
such as getNumFailedAppAttempts also wont be exact number since attempt failure is taken from

> RM might fail to restart when recovering apps whose attempts are missing
> ------------------------------------------------------------------------
>                 Key: YARN-4497
>                 URL: https://issues.apache.org/jira/browse/YARN-4497
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Jun Gong
>            Assignee: Jun Gong
>            Priority: Critical
>         Attachments: YARN-4497.01.patch
> Find following problem when discussing in YARN-3480.
> If RM fails to store some attempts in RMStateStore, there will be missing attempts in
RMStateStore, for the case storing attempt1, attempt2 and attempt3, RM successfully stored
attempt1 and attempt3, but failed to store attempt2. When RM restarts, in *RMAppImpl#recover*,
we recover attempts one by one, for this case, we will recover attmept1, then attempt2. When
recovering attempt2, we call  *((RMAppAttemptImpl)this.currentAttempt).recover(state)*, it
will first find its ApplicationAttemptStateData, but it could not find it, an error will come
at *assert attemptState != null*(*RMAppAttemptImpl#recover*, line 880).

This message was sent by Atlassian JIRA

View raw message