From yarn-issues-return-32856-apmail-hadoop-yarn-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 1 21:11:40 2014 Return-Path: X-Original-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5AC8B11E46 for ; Fri, 1 Aug 2014 21:11:40 +0000 (UTC) Received: (qmail 88139 invoked by uid 500); 1 Aug 2014 21:11:40 -0000 Delivered-To: apmail-hadoop-yarn-issues-archive@hadoop.apache.org Received: (qmail 88103 invoked by uid 500); 1 Aug 2014 21:11:40 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: yarn-issues@hadoop.apache.org Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 88092 invoked by uid 99); 1 Aug 2014 21:11:40 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Aug 2014 21:11:40 +0000 Date: Fri, 1 Aug 2014 21:11:40 +0000 (UTC) From: "Jian He (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (YARN-1372) Ensure all completed containers are reported to the AMs across RM restart MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-1372?page=3Dcom.atlassian.= jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D14082= 967#comment-14082967 ]=20 Jian He commented on YARN-1372: ------------------------------- Thanks for working on this. Scan through the patch: - If AM never pulls the containers (e.g. AM crashed), the containers will r= emain on NM forever ? - NM-RM protocol changes: The patch adds a new getCleanedupContainersNotifi= ed method and some associated event/state machine changes.Found that maybe = we can just use NodeHeartbeatResponse#getContainersToCleanup to notify NM t= o remove the containers from the context. the newly added getCleanedupConta= inersNotified() is very similar to the getContainersToCleanup. The code pat= h in RM seems also similar.=20 > Ensure all completed containers are reported to the AMs across RM restart > ------------------------------------------------------------------------- > > Key: YARN-1372 > URL: https://issues.apache.org/jira/browse/YARN-1372 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager > Reporter: Bikas Saha > Assignee: Anubhav Dhoot > Attachments: YARN-1372.prelim.patch > > > Currently the NM informs the RM about completed containers and then remov= es those containers from the RM notification list. The RM passes on that co= mpleted container information to the AM and the AM pulls this data. If the = RM dies before the AM pulls this data then the AM may not be able to get th= is information again. To fix this, NM should maintain a separate list of su= ch completed container notifications sent to the RM. After the AM has pulle= d the containers from the RM then the RM will inform the NM about it and th= e NM can remove the completed container from the new list. Upon re-register= with the RM (after RM restart) the NM should send the entire list of compl= eted containers to the RM along with any other containers that completed wh= ile the RM was dead. This ensures that the RM can inform the AM's about all= completed containers. Some container completions may be reported more than= once since the AM may have pulled the container but the RM may die before = notifying the NM about the pull. -- This message was sent by Atlassian JIRA (v6.2#6252)