Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Fri, 12 Sep 2014 23:25:34 +0000 (UTC)
From: "Jian He (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12676565.1383116589000.24194.1410564334695@Atlassian.JIRA>
In-Reply-To: <JIRA.12676565.1383116589000@Atlassian.JIRA>
References: <JIRA.12676565.1383116589000@Atlassian.JIRA>
 <JIRA.12676565.1383116589025@arcas>
Subject: [jira] [Commented] (YARN-1372) Ensure all completed containers are
 reported to the AMs across RM restart
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/YARN-1372?page=3Dcom.atlassian.=
jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D14132=
294#comment-14132294 ]=20

Jian He commented on YARN-1372:
-------------------------------

bq. Why not ack the finishedContainersSentToAM containers to NM in the Base=
FinalTransition and then call clear on finishedContainersSentToAM.
Because AppAttemptRemovedSchedulerEvent is sent in the same transition to s=
top all the running containers, which then sends finishedContainers back to=
 RMAppAttempt. So by this time, RMAppAttempt has not yet collected the fini=
shedContainers.=20

> Ensure all completed containers are reported to the AMs across RM restart
> -------------------------------------------------------------------------
>
>                 Key: YARN-1372
>                 URL: https://issues.apache.org/jira/browse/YARN-1372
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Bikas Saha
>            Assignee: Anubhav Dhoot
>         Attachments: YARN-1372.001.patch, YARN-1372.001.patch, YARN-1372.=
002_NMHandlesCompletedApp.patch, YARN-1372.002_RMHandlesCompletedApp.patch,=
 YARN-1372.002_RMHandlesCompletedApp.patch, YARN-1372.003.patch, YARN-1372.=
004.patch, YARN-1372.005.patch, YARN-1372.005.patch, YARN-1372.prelim.patch=
, YARN-1372.prelim2.patch
>
>
> Currently the NM informs the RM about completed containers and then remov=
es those containers from the RM notification list. The RM passes on that co=
mpleted container information to the AM and the AM pulls this data. If the =
RM dies before the AM pulls this data then the AM may not be able to get th=
is information again. To fix this, NM should maintain a separate list of su=
ch completed container notifications sent to the RM. After the AM has pulle=
d the containers from the RM then the RM will inform the NM about it and th=
e NM can remove the completed container from the new list. Upon re-register=
 with the RM (after RM restart) the NM should send the entire list of compl=
eted containers to the RM along with any other containers that completed wh=
ile the RM was dead. This ensures that the RM can inform the AM's about all=
 completed containers. Some container completions may be reported more than=
 once since the AM may have pulled the container but the RM may die before =
notifying the NM about the pull.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)