Return-Path: X-Original-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D3848195E0 for ; Thu, 17 Mar 2016 23:55:33 +0000 (UTC) Received: (qmail 36287 invoked by uid 500); 17 Mar 2016 23:55:33 -0000 Delivered-To: apmail-hadoop-yarn-issues-archive@hadoop.apache.org Received: (qmail 36230 invoked by uid 500); 17 Mar 2016 23:55:33 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: yarn-issues@hadoop.apache.org Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 36218 invoked by uid 99); 17 Mar 2016 23:55:33 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2016 23:55:33 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 7021D2C1F58 for ; Thu, 17 Mar 2016 23:55:33 +0000 (UTC) Date: Thu, 17 Mar 2016 23:55:33 +0000 (UTC) From: "Haibo Chen (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (YARN-4766) NM should not aggregate logs older than the retention policy MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200655#comment-15200655 ] Haibo Chen commented on YARN-4766: ---------------------------------- fixed the liscense + checkstyle issues. The unit test failure is unrelated to the patch. I have create another jira to fix the test failure at https://issues.apache.org/jira/browse/YARN-4838 > NM should not aggregate logs older than the retention policy > ------------------------------------------------------------ > > Key: YARN-4766 > URL: https://issues.apache.org/jira/browse/YARN-4766 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation, nodemanager > Reporter: Haibo Chen > Assignee: Haibo Chen > Attachments: yarn4766.001.patch, yarn4766.002.patch > > > When a log aggregation fails on the NM the information is for the attempt is kept in the recovery DB. Log aggregation can fail for multiple reasons which are often related to HDFS space or permissions. > On restart the recovery DB is read and if an application attempt needs its logs aggregated, the files are scheduled for aggregation without any checks. The log files could be older than the retention limit in which case we should not aggregate them but immediately mark them for deletion from the local file system. -- This message was sent by Atlassian JIRA (v6.3.4#6332)